python爬蟲 requests模組

功能：模擬瀏覽器發請求

指定url：

發起請求：

獲取響應資料：

持久化儲存：

import requests

# 指定url
url =
''# 發起請求，返回乙個響應對像
response=requests.get(url=url)
# 獲取響應資料：text返回的是字串型別的響應資料
page_text = response.text
# 持久化儲存
with
open
('./bing.html'
,'w'
,encoding=
'utf-8'
)as f:
f.write(page_text)

import requests

# 指定url
url =
''headers =
# 處理url攜帶的引數：封裝到字典中
kw =
input
('enter a word:'
)param =
# 發起請求
response = requests.get(url=url, headers=headers, params=param)
# 獲取響應資料
page_text = response.text
# 持久化儲存
file_name = kw +
'.html'
with
open
(file_name,
'w', encoding=
'utf-8'
)as f:
f.write(page_text)

import requests

import json
# 指定url
url =
''headers =
kw =
input
('input your keyword:'
)data =
response = requests.post(url=url,data=data,headers=headers)
dic_obj = response.json(
)fp =
open
('./dog.json'
,'w'
,encoding=
'utf-8'
)json.dump(dic_obj,fp=fp,ensure_ascii=
false
)

import requests

import json
url =
''headers =
param =
list_data = requests.get(url=url,headers=headers,params=param)
.json(
)fp =
open
('./douban.json'
,mode=
'w',encoding=
'utf-8'
)json.dump(list_data,fp=fp,ensure_ascii=
false
)

import requests

import json
url =
''headers =
data =
page_json = requests.post(url=url, headers=headers, data=data)
.json(
)fp =
open
('./kfc.json'
,mode=
'w',encoding=
'utf-8'
)json.dump(page_json,fp=fp,ensure_ascii=
false
)

import requests

import json
url =
''headers =
data =
# 獲取所有id
data_obj = requests.post(url=url, data=data, headers=headers)
.json(
)id_list =
for i in data_obj[
'list']:
'id'])
# 獲取每個id所對應的資料
all_data_list =
url_byld =
''forid
in id_list:
data_id =
detail_json = requests.post(url=url_byld,
headers=headers,
data=data_id)
.json(
)# 持久化儲存
fp =
open
('./alldata.json'
,mode=
'w',encoding=
'utf-8'
)json.dump(all_data_list,fp=fp,ensure_ascii=
false
)

Python爬蟲 Request模組

文章說明了request模組的意義，且強調了request模組使用更加方便。接下來介紹幾種常用的request操作，並且會在後續補充說明一些特定用法。匯入檔案 import requests一 請求 右邊為請求語句，返回值為response回應 r requests.get r requests.p...

python爬蟲利器 request庫

request庫比urllib2庫更為高階，因為其功能更強大，更易於使用。使用該庫可以十分方便我們的抓取。基本請求 r requests.get r requests.post r requests.put r requests.delete r requests.head r requests.o...

爬蟲 python（二）初識request

from urllib.request import urlopen 傳送請求，獲取伺服器給的響應 url response urlopen url 讀取結果,無法正常顯示中文 html response.read 進行解碼操作，轉為utf 8 html decode html.decode 列印結...