首页 > 编程语言 > 详细

python爬虫知识点三--解析豆瓣top250数据

时间:2017-11-19 17:56:17      阅读:366      评论:0      收藏:0      [点我收藏+]
一。利用cookie访问
import
requests headers = {User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36} cookies = {cookie: bid=a3MhK2YEpZw; ll="108296"; ps=y; ue="t.t.panda@hotmail.com"; _pk_ref.100001.8cb4=%5B%22%22%2C%22%22%2C1482650884%2C%22https%3A%2F%2Fwww.so.com%2Fs%3Fie%3Dutf-8%26shb%3D1%26src%3Dhome_so.com%26q%3Dpython%2B%25E8%25B1%2586%25E7%2593%25A3%25E6%25BA%2590%22%5D; _gat_UA-7019765-1=1; ap=1; __utmt=1; _ga=GA1.2.1329310863.1477654711; dbcl2="2625855:/V89oXS4WD4"; ck=EePo; push_noty_num=0; push_doumail_num=0; _pk_id.100001.8cb4=40c3cee75022c8e1.1477654710.8.1482652441.1482639716.; _pk_ses.100001.8cb4=*; __utma=30149280.1329310863.1477654711.1482643456.1482650885.10; __utmb=30149280.19.10.1482650885; __utmc=30149280; __utmz=30149280.1482511651.7.6.utmcsr=blog.csdn.net|utmccn=(referral)|utmcmd=referral|utmcct=/alanzjl/article/details/50681289; __utmv=30149280.262; _vwo_uuid_v2=64E0E442544CB2FE2D322C59F01F1115|026be912d24071903cb0ed891ae9af65} url = http://www.douban.com r = requests.get(url, cookies = cookies, headers = headers) with open(douban_2.txt, wb+) as f: f.write(r.content)

二。利用Xpath搜索

import requests
from lxml import etree

s = requests.Session()
for id in range(0, 251, 25):
    print (id)
技术分享图片
    url = https://movie.douban.com/top250/?start- + str(id)
    r = s.get(url)
    r.encoding = utf-8
    root = etree.HTML(r.content)
    items = root.xpath(‘//ol/li/div[@class="item"]‘) //利用xpath的标签选择
技术分享图片

 



# print(len(items)) for item in items: title = item.xpath(./div[@class="info"]//a/span[@class="title"]/text())//如下找到中文名字
技术分享图片

 


        name = title[0].encode(gb2312, ignore).decode(gb2312)//title是一个数组  先encoding 再decode确保字符不混在一起
        # rank = item.xpath(‘./div[@class="pic"]/em/text()‘)[0]
        rating = item.xpath(.//div[@class="bd"]//span[@class="rating_num"]/text())[0]
技术分享图片

 

print(name, rating)


 结果:成功爬取前250个评分技术分享图片

 

 ps:必须知道网页的结构

python爬虫知识点三--解析豆瓣top250数据

原文:http://www.cnblogs.com/LHWorldBlog/p/7860506.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!