爬蟲每日一練(1)

2021-08-16 05:33:35 字數 1973 閱讀 6212

教程:

爬取乙個套圖(加了防盜煉解決方法)

#coding=utf-8

import requests

from bs4 import beautifulsoup

import os

import sys

if(os.name == 'nt'):

print(u'你正在使用win平台')

else:

print(u'你正在使用linux平台')

'referer':''}#偽裝瀏覽器

all_url = ''

start_html = requests.get(all_url,headers = header)

path = 'f:/meizi/'#位址

soup = beautifulsoup(start_html.text,'html.parser')

page = soup.find_all('a',class_='page-numbers')

max_page = page[-2].text

same_url = '/page/'

for n in range(1,int(max_page) + 1):

ul = same_url+str(n)

start_html = requests.get(ul,headers = header)

soup = beautifulsoup(start_html.text,"html.parser")

all_a = soup.find('div',class_='postlist').find_all('a',target = '_blank')

for a in all_a:

title = a.get_text()

if(title !=''):

print("準備爬取:"+title)

if(os.path.exists(path+title.strip().replace('?',''))):

flag=1

else:

os.makedirs(path+title.strip().replace('?',''))

flag=0

os.chdir(path + title.strip().replace('?',''))

href = a['href']

html = requests.get(href,headers = header)

mess = beautifulsoup(html.text,"html.parser")

pic_max = mess.find_all('span')

pic_max = pic_max[10].text

if(flag == 1 and len(os.listdir(path+title.strip().replace('?',''))) >= int(pic_max)):

print('已儲存完畢,跳過')

continue

for num in range(1,int(pic_max)+1):

pic = href+'/'+str(num)

html = requests.get(pic,headers = header)

mess = beautifulsoup(html.text,"html.parser")

pic_url = mess.find('img',alt = title)

html = requests.get(pic_url['src'],headers = header)

file_name = pic_url['src'].split(r'/')[-1]

f = open(file_name,'wb')

f.write(html.content)

f.close()

print('完成')

print('第',n,'頁完成')

每日一練 1

左連線 以左表為主,右表沒有的填null 公升序 倒序 order by預設公升序 order by xx desc 取最大值 max 函式,例如 select max hire date from employees limit 1,2和limit 1offset 2的含義 limit 1,2第乙...

每日一練 no 1

問題 四個數字1 2 3 4 能組成多少個互不相同且無重複的三位數?解答方法一 先不考慮重複將個位 十位 百位填入數值,得到後判斷各個數字上是否有重複,棄去重複項 實現 total for i in range 1 5 for j in range 1 5 for k in range 1 5 if...

每日一練day 1

1 關鍵字static 的作用?static關鍵字至少有下列n個作用 1 設定變數的儲存域,函式體內static變數的作用範圍為該函式體,不同於auto變數,該變數的記憶體只被分配一次,因此其值在下次呼叫時仍維持上次的值 2 限制變數的作用域,在模組內的static全域性變數可以被模組內所用函式訪問...