爬虫基础之Request类

categories:

• 爬虫
  tags:
• Request

• 模拟请求拉钩网

request.Request类

如果想要在请求的时候增加一些请求头，用request.Request

from urllib import request
headers = {
    'userAgent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
req = request.Request("http://www.baidu.com/",headers = headers)
resp = request.urlopen(req)
print(resp.read()[:10])
#返回结果如下
b'<!DOCTYPE '

模拟请求拉钩网站

from urllib import request,parse
headers = {
    'userAgent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',
    'Referer':'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
}
url = 'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput='
data = {
    'first':'true',
    'pn':1,
    'kd':'python'
}
req = request.Request(url,headers=headers,data=parse.urlencode(data).encode('utf-8'),method='POST')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))

爬虫基础之Request类

原文：https://www.cnblogs.com/anyux/p/12203062.html