如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題
這篇文章將為大家詳細(xì)講解有關(guān)如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題，小編覺(jué)得挺實(shí)用的，因此分享給大家做個(gè)參考，希望大家閱讀完這篇文章后可以有所收獲。
我們提供的服務(wù)有：網(wǎng)站建設(shè)、成都做網(wǎng)站、微信公眾號(hào)開(kāi)發(fā)、網(wǎng)站優(yōu)化、網(wǎng)站認(rèn)證、焦作ssl等。為上千多家企事業(yè)單位解決了網(wǎng)站和推廣的問(wèn)題。提供周到的售前咨詢和貼心的售后服務(wù)，是有科學(xué)管理、有技術(shù)的焦作網(wǎng)站制作公司
一般情況下抓取網(wǎng)頁(yè)時(shí)找到url中可以從 post 請(qǐng)求中獲取數(shù)據(jù)，但得到的響應(yīng)是400而不是 200時(shí)，可以嘗試從服務(wù)器獲得正確的響應(yīng)，使用json=參數(shù)requests進(jìn)行嘗試可以解決。
問(wèn)題案例：
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0', 'Accept': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'Content-Type': 'application/x-www-form-urlencoded', 'Origin': 'https://www.ketto.org', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'Referer': 'https://www.ketto.org/', 'Connection': 'keep-alive',
}
params = (
('x-algolia-agent', 'Algolia for JavaScript (3.35.1); Browser (lite); angular (8.2.14); angular-instantsearch (3.0.0-beta.4); instantsearch.js (3.7.0); JS Helper (2.28.1)'),
('x-algolia-application-id', 'NN2UORRIZX'),
('x-algolia-api-key', 'b2caa1b0589e8db9398d5fe2a40bbaed'),
)
data = [
('{requests:[{indexName:fundraiser_prod,params:query', ''),
('hitsPerPage', '9'),
('hitsPerPage', '1'),
('hitsPerPage', '1'),
('hitsPerPage', '1'),
('maxValuesPerFacet', '10'),
('maxValuesPerFacet', '10'),
('maxValuesPerFacet', '10'),
('maxValuesPerFacet', '10'),
('page', '1'),
('page', '0'),
('page', '0'),
('page', '0'),
('highlightPreTag', '__ais-highlight__'),
('highlightPreTag', '__ais-highlight__'),
('highlightPreTag', '__ais-highlight__'),
('highlightPreTag', '__ais-highlight__'),
('highlightPostTag', '__/ais-highlight__'),
('highlightPostTag', '__/ais-highlight__'),
('highlightPostTag', '__/ais-highlight__'),
('highlightPostTag', '__/ais-highlight__'),
('facets', '["cause.label","tags","address"]'),
('facets', '["cause.label"]'),
('facets', '["tags"]'),
('facets', '["address"]'),
('tagFilters', ''),
('tagFilters', ''),
('tagFilters', ''),
('tagFilters', ''),
('facetFilters', '[["cause.label:"],["tags:"],["address:"]]},{indexName:fundraiser_prod,params:query='),
('facetFilters', '[["tags:"],["address:"]]},{indexName:fundraiser_prod,params:query='),
('facetFilters', '[["cause.label:"],["address:"]]},{indexName:fundraiser_prod,params:query='),
('facetFilters', '[["cause.label:"],["tags:"]]}]}'),
('attributesToRetrieve', '[]'),
('attributesToRetrieve', '[]'),
('attributesToRetrieve', '[]'),
('attributesToHighlight', '[]'),
('attributesToHighlight', '[]'),
('attributesToHighlight', '[]'),
('attributesToSnippet', '[]'),
('attributesToSnippet', '[]'),
('attributesToSnippet', '[]'),
('analytics', 'false'),
('analytics', 'false'),
('analytics', 'false'),
('clickAnalytics', 'false'),
('clickAnalytics', 'false'),
('clickAnalytics', 'false'),
]
response = requests.post('https://nn2uorrizx-dsn.algolia.net/1/indexes/*/queries', headers=headers, params=params, data=data)
print(response)代碼進(jìn)行優(yōu)化處理：
import requestsurl = "https://nn2uorrizx-dsn.algolia.net/1/indexes/*/queries"params = {
"x-algolia-agent": "Algolia for JavaScript (3.35.1); Browser (lite); angular (8.2.14); angular-instantsearch (3.0.0-beta.4); instantsearch.js (3.7.0); JS Helper (2.28.1)", "x-algolia-application-id": "NN2UORRIZX", "x-algolia-api-key": "b2caa1b0589e8db9398d5fe2a40bbaed",}data = {
"requests": [
{
"indexName": "fundraiser_prod", "params": "query=&hitsPerPage=9&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&facets=%5B%22cause.label%22%2C%22tags%22%2C%22address%22%5D&tagFilters=&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22tags%3A%22%5D%2C%5B%22address%3A%22%5D%5D",
},
{
"indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22cause.label%22%5D&facetFilters=%5B%5B%22tags%3A%22%5D%2C%5B%22address%3A%22%5D%5D",
},
{
"indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22tags%22%5D&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22address%3A%22%5D%5D",
},
{
"indexName": "fundraiser_prod", "params": "query=&hitsPerPage=1&maxValuesPerFacet=10&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=%5B%22address%22%5D&facetFilters=%5B%5B%22cause.label%3A%22%5D%2C%5B%22tags%3A%22%5D%5D",
},
]}data = requests.post(url, params=params, json=data).json()print(json.dumps(data, indent=4))打印結(jié)果：
{
"results": [
{
"hits": [
{
"id": 180773,
"title": "Feeding From Far-Ration Distribution Amid Lockdown",
"end_date": "2021-09-11 23:59:59",
"amount_requested": 65000000,
"entity_details_id": 1505699,
"creator_entity_details_id": 1463388,
"address_1": "Mumbai",
"creation_date": "2020-04-05 15:10:12",
"parent_cause_id": 48,
"event_entity_details_id": 1399904,
"sucess_story_flag": 0,
"custom_tag": "FeedingFromFarForCorona",
"beneficiary": {
"full_name": "Prayatna "
},
"raised": {
"campaign_id": 180773,
"backers": 10616,
"raised": "45094771.40",
"usdraised": "644211.02",
"currencies": {
"INR": 45094771,
"USD": 644211,
"GBP": 501053,
"EUR": 536843,
"AED": 2254739,
"SGD": 867207,
"SAR": 2254739
}
},
"widget": {
"media_type": "image",
"file_name": "wid60a133a89542c.jpg",
"path": "/media/campaign/180000/180773/image/",
"cdn_path": "https://d1vdjc70h9nzd9.cloudfront.net/media/campaign/180000/180773/image/wid60a133a89542c.jpg"
},
"cause": {
"info_1": "Food & Hunger",
"info_3": "Orange",
"label": "Food & Hunger"
},
"campaigner": {
"id": 1463388,
"fname": "Pooja Reddy",
"lname": null,
"full_name": "Pooja Reddy ",
"disable_foreign_donation": 0,
"no_80g": false,
"user_details_id": 1470493,
"entity_type": "individual",
"tax_benefit": false,
"avtar": {
"entity_type_id": 1463388,
"entity_type": "individual",
"media_type": "image",
"file_name": "60efdb6db4758.jpg",
"path": "/media/individual/1463000/1463388/image/",
"cdn_path": "https://d1vdjc70h9nzd9.cloudfront.net/media/individual/1463000/1463388/image/60efdb6db4758.jpg"
}
},
"taxexempted": {
"id": 1505699,
"no_80g": true,
"entity_type": "ngo",
"tax_benefit": true
},... and so on.}關(guān)于“如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題”這篇文章就分享到這里了，希望以上內(nèi)容可以對(duì)大家有一定的幫助，使各位可以學(xué)到更多知識(shí)，如果覺(jué)得文章不錯(cuò)，請(qǐng)把它分享出去讓更多的人看到。
本文名稱：如何解決Python爬取網(wǎng)頁(yè)時(shí)請(qǐng)求中收到響應(yīng)400的問(wèn)題
分享路徑：http://chinadenli.net/article2/gsjeoc.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供靜態(tài)網(wǎng)站、用戶體驗(yàn)、做網(wǎng)站、微信公眾號(hào)、網(wǎng)站營(yíng)銷、品牌網(wǎng)站建設(shè)
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)
- 搜索引擎優(yōu)化與用戶體驗(yàn)的關(guān)系是怎樣的 2023-01-16
- 網(wǎng)店商城建站如何提高用戶體驗(yàn)？ 2021-02-01
- 網(wǎng)站建設(shè)中從五個(gè)方面提升網(wǎng)站用戶體驗(yàn) 2022-06-12
- 你所認(rèn)為的“用戶體驗(yàn)”，真的符合搜索引擎之百度排名？ 2014-06-26
- 成都網(wǎng)絡(luò)推廣如何提升用戶體驗(yàn)？ 2022-09-26
- 淺談良好用戶體驗(yàn)的要素 2021-08-19
- 電商網(wǎng)站建設(shè)：提高用戶體驗(yàn)的六大技巧 2013-11-10
- 如何完善你的UX用戶體驗(yàn) 2016-08-09
- 企業(yè)網(wǎng)站應(yīng)該怎么展示產(chǎn)品的方法及用戶體驗(yàn) 2016-11-10
- 沒(méi)有與用戶體驗(yàn)的網(wǎng)站相當(dāng)于一個(gè)廢品網(wǎng)站 2016-10-28
- 網(wǎng)站優(yōu)化之用戶體驗(yàn) 2021-04-19
- 上海網(wǎng)站制作開(kāi)發(fā)離不開(kāi)用戶體驗(yàn) 2020-11-23