豆瓣书评爬虫

豆瓣书评爬虫

一年前学的爬虫,忘得差不多了,现在复习一下
最近读了《白夜行》,就拿它下手喽

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import requests
from lxml import etree
import pandas as pd
import time

data = []
def get_userdata(page):
for i in range(1, page + 1):
url = 'https://book.douban.com/subject/3259440/comments/hot?p={}'.format(i)
print("正在爬取第{}页\n".format(i))
print(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
time.sleep(1)
r = requests.get(url)
s = etree.HTML(r.text)

xiaodata = s.xpath('//div[@id="comments"]//div/p/text()')

data.extend(xiaodata)

if __name__ == '__main__':
get_userdata(20)
df = pd.DataFrame(data)
df.to_csv('白夜行.csv', encoding='utf-8-sig')
print("任务完成")

最后结果,爬20页意思意思就好啦

-------------End Of This ArticleThank You For Reading-------------