有小夥伴說想拿鏈家二手房資訊做資料分析，讓幫忙抓點資料。並沒有搞過，網上搜了一些資料試了一下，感覺不難可以搞，下面小結一下。
三方庫庫的安裝也比較簡單，直接使用 pip install 相應的庫名 即可：
pip install pandas
pip install requests
pip install beautifulsoup4
複製**
...00
複製**
複製**
# 我這裡設定為3秒
time.sleep(3)
複製**
import requests
from bs4 import beautifulsoup
import sys
import os
import time
import pandas as pd
import numpy as np
from parsel import selector
import re
headers =
def catchhouselist(url):
resp = requests.get(url, headers=headers, stream=true)
if resp.status_code == 200:
reg = re.compile('.*?)
urls = re.findall(reg, resp.text)
return urls
return
def catchhousedetail(url):
resp = requests.get(url, headers=headers)
print(url)
if resp.status_code == 200:
info = {}
soup = beautifulsoup(resp.text, 'html.parser')
info['標題'] = soup.select('.main')[0].text
info['總價'] = soup.select('.total')[0].text
info['總價單位'] = soup.select('.unit')[0].text
info['每平方售價'] = soup.select('.unitpricevalue')[0].text
# p = soup.select('.tax')
# info['參考總價'] = soup.select('.tax')[0].text
info['建造時間'] = soup.select('.subinfo')[2].text
info['小區名稱'] = soup.select('.info')[0].text
info['所在區域'] = soup.select('.info a')[0].text + ':' + soup.select('.info a')[1].text
info['鏈家編號'] = str(url)[34:].rsplit('.html')[0]
info['房屋戶型'] = str(soup.select('.content')[2].select('.label')[0].next_sibling)
info['所在樓層'] = soup.select('.content')[2].select('.label')[1].next_sibling
info['建築面積'] = soup.select('.content')[2].select('.label')[2].next_sibling
info['戶型結構'] = soup.select('.content')[2].select('.label')[3].next_sibling
info['套內面積'] = soup.select('.content')[2].select('.label')[4].next_sibling
info['建築型別'] = soup.select('.content')[2].select('.label')[5].next_sibling
info['房屋朝向'] = soup.select('.content')[2].select('.label')[6].next_sibling
info['建築結構'] = soup.select('.content')[2].select('.label')[7].next_sibling
info['裝修情況'] = soup.select('.content')[2].select('.label')[8].next_sibling
info['梯戶比例'] = soup.select('.content')[2].select('.label')[9].next_sibling
info['供暖方式'] = soup.select('.content')[2].select('.label')[10].next_sibling
info['配備電梯'] = soup.select('.content')[2].select('.label')[11].next_sibling
# info['產權年限'] = str(soup.select('.content')[2].select('.label')[12].next_sibling)
return info
pass
filename = './鏈家二手房.xlsx'
dfnew = pd.dataframe([info])
if(os.path.exists(filename)):
sheet = pd.read_excel(filename)
dfold = pd.dataframe(sheet)
df = pd.concat([dfold, dfnew])
df.to_excel(filename)
else:
dfnew.to_excel(filename)
def catch():
pages = [''.format(x) for x in range(1, 1001)]
for page in pages:
print(page)
houselisturls = catchhouselist(page)
for housedetailurl in houselisturls:
try:
info = catchhousedetail(housedetailurl)
except:
pass
time.sleep(3)
pass
if __name__ == '__main__':
catch()
複製**
Python爬取鏈家二手房資訊
2 資料庫表結構 使用物件導向的方式，搭建專案框架 import requests from bs4 import beautifulsoup import pymysql class lianjiaspider mydb pymysql.connect localhost root 123456 ...
python爬蟲爬取鏈家二手房資訊
問題一 鏈家 也有反爬蟲策略和robots限制，robots限制忽略 不然沒法爬 另外頻繁爬取會直接導致被ban，需要隔天才會解禁止。防止被ban的方法有多種，1.禁止cookie 2.設定header 3.加大爬取間隔 4.使用 我只用了前三種方法，具體可以在settings.py 和middle...
python爬取鏈家網二手房資訊
朋友請我幫忙做的期末作業，我自己不是愛說話，直接分享 可以直接執行的，期中用的是 python 3.6版本，導包的時候直接在cmd裡面用的pip install 包名，其中有的包安裝失敗，提示pip需要公升級，可以看一下這個鏈結 下面是 在這裡插入 片 usr bin env python3 cod...