在爬取这个网站的时候，如果直接用requests进行请求，得到的HTML并没有目标数据，所以我估计这不是静态网页，而是动态网页，于是就到XHR查看了一下，果真，那些数据是在如图所示的链接中：
1 import requests 2 import json 3 headers = { 4 ‘User-Agent‘: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36‘, 5 ‘Cookie‘: ‘pgv_info=ssid=s9071181272; ts_last=news.qq.com/; pgv_pvid=7613784555; ts_uid=465011805; ad_play_index=77; pac_uid=0_8ed4c1084a611‘, 6 } 7 url = ‘https://i.news.qq.com/trpc.qqnews_web.kv_srv.kv_srv_http_proxy/list?sub_srv_id=24hours&srv_id=pc&offset=0&limit=20&strategy=1&ext={%22pool%22:[%22top%22],%22is_filter%22:7,%22check_type%22:true}‘ 8 9 10 news_dic = requests.get(url = url, headers = headers,data = data).json()[‘data‘][‘list‘] 11 # print(type(news_dic)) 12 # print(news_dic) 13 14 for new in news_dic: 15 print({‘标题‘:new[‘title‘],‘链接‘:new[‘url‘]})
原文：https://www.cnblogs.com/industrial-fd-2019/p/14725649.html