引擎（Engine）
爬虫程序（Spider）
调度器（Scheduler）
下载器（Downloader）
管道文件（Pipeline）
# 两个中间件
下载器中间件（Downloader Middlewares）
蜘蛛中间件（Spider Middlewares）
1、Engine向Spider索要URL,交给Scheduler入队列
2、Scheduler处理后出队列,通过Downloader Middlewares交给Downloader去下载
3、Downloader得到响应后,通过Spider Middlewares交给Spider
4、Spider数据提取：
1、数据交给Pipeline处理
2、需要跟进URL,继续交给Scheduler入队列，依次循环
# 创建爬虫项目
scrapy startproject 项目名
# 创建爬虫文件
cd 项目文件夹
scrapy genspider 爬虫名 域名
# 运行爬虫
scrapy crawl 爬虫名
1、scrapy startproject Tencent
2、cd Tencent
3、scrapy genspider tencent tencent.com
4、items.py(定义爬取数据结构)
import scrapy
class TencentItem(scrapy.Item):
job_name = scrapy.Field()
5、tencent.py（写爬虫文件）
import scrapy
class TencentSpider(scarpy.Spider):
name = ‘tencent‘
allowed_domains = [‘tencent.com‘]
start_urls = [‘http://tencent.com/‘]
def parse(self,response):
xxx
yield item
6、pipelines.py(数据处理)
class TencentPipeline(object):
def process_item(self,item,spider):
return item
7、settings.py(全局配置)
LOG_LEVEL = ‘‘
LOG_FILE = ‘‘
FEED_EXPORT_ENCODING = ‘‘
8、终端：scrapy crawl tencent
# 属性
1、response.text ：获取响应内容 - 字符串
2、response.body ：获取bytes数据类型
3、response.xpath(‘‘)
# response.xpath(‘‘)调用方法
1、结果 ：列表,元素为选择器对象
# <selector xpath=‘//article‘ data=‘‘>
2、.extract() ：提取文本内容,将列表中所有元素序列化为Unicode字符串
3、.extract_first() ：提取列表中第1个文本内容
4、.get() ： 提取列表中第1个文本内容
从爬虫文件(spider)的start_urls变量中遍历URL地址，把下载器返回的响应对象（response）交给爬虫文件的parse()函数处理
# start_urls = [‘http://www.baidu.com/‘]
重写start_requests()方法，从此方法中获取URL，交给指定的callback解析函数处理
1、去掉start_urls变量
2、def start_requests(self):
# 生成要爬取的URL地址，利用scrapy.Request()方法交给调度器
DEBUG < INFO < WARNING < ERROR < CRITICAL
1、在setting.py中定义相关变量
2、pipelines.py中新建管道类，并导入settings模块
def open_spider(self,spider):
# 爬虫开始执行1次,用于数据库连接
def process_item(self,item,spider):
# 用于处理抓取的item数据
def close_spider(self,spider):
# 爬虫结束时执行1次,用于断开数据库连接
3、settings.py中添加此管道
ITEM_PIPELINES = {‘‘:200}
# 注意 ：process_item() 函数中一定要 return item
scrapy crawl maoyan -o maoyan.csv
scrapy crawl maoyan -o maoyan.json
# settings.py FEED_EXPORT_ENCODING = ‘utf-8‘
# 1、设置日志级别
LOG_LEVEL = ‘‘
# 2、保存到日志文件(不在终端输出)
LOG_FILE = ‘‘
# 3、设置数据导出编码(主要针对于json文件)
FEED_EXPORT_ENCODING = ‘utf-8‘
# 4、非结构化数据存储路径
IMAGES_STORE = ‘‘
FILES_STORE = ‘‘
# 5、设置User-Agent
USER_AGENT = ‘‘
# 6、设置最大并发数(默认为16)
CONCURRENT_REQUESTS = 32
# 7、下载延迟时间(每隔多长时间请求一个网页)
DOWNLOAD_DELAY = 0.5
# 8、请求头
DEFAULT_REQUEST_HEADERS = {}
# 9、添加项目管道
ITEM_PIPELINES = {‘项目名.pipelines.类名‘:200}
# 10、添加下载器中间件
DOWNLOADER_MIDDLEWARES = {‘项目名.middlewares.类名‘:200}
# 11、cookie(默认禁用,取消注释-True|False都为开启)
COOKIES_ENABLED = False
1、url
2、callback
3、meta ：传递数据,定义代理
meta = {‘name‘:name,‘proxy‘:‘http://IP:Port‘}
scrapy startproject Tencent
cd Tencent
scrapy genspider tencent hr.tencent.com
# 一级页面(postId):
https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1566266592644&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=&keyword={}&pageIndex={}&pageSize=10&language=zh-cn&area=cn
# 二级页面
https://careers.tencent.com/tencentcareer/api/post/ByPostId?timestamp=1566266695175&postId={}&language=zh-cn
# 名称+类别+职责+要求+地址+时间
job_name = scrapy.Field()
job_type = scrapy.Field()
job_duty = scrapy.Field()
job_require = scrapy.Field()
job_address = scrapy.Field()
job_time = scrapy.Field()
import scrapy
from urllib import parse
import requests
import json
from ..items import TencentItem
class TencentSpider(scrapy.Spider):
name = ‘tencent‘
allowed_domains = [‘careers.tencent.com‘]
one_url = ‘https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1566266592644&countryId=&cityId=&bgIds=&productId=&categoryId=&parentCategoryId=&attrId=&keyword={}&pageIndex={}&pageSize=10&language=zh-cn&area=cn‘
two_url = ‘https://careers.tencent.com/tencentcareer/api/post/ByPostId?timestamp=1566266695175&postId={}&language=zh-cn‘
headers = { ‘User-Agent‘:‘Mozilla/5.0‘ }
# 重写start_requests()
keyword = input(‘请输入职位类别:‘)
keyword = parse.quote(keyword)
def start_requests(self):
total = self.get_total()
# 生成一级页面所有页的URL地址,交给调度器
for index in range(1,total+1):
url = self.one_url.format(self.keyword,index)
yield scrapy.Request(
url=url,
callback=self.parse_one_page
)
# 获取总页数
def get_total(self):
url = self.one_url.format(self.keyword, 1)
html = requests.get(url=url, headers=self.headers).json()
count = html[‘Data‘][‘Count‘]
if count % 10 == 0:
total = count // 10
else:
total = (count // 10) + 1
return total
# 一级页面:提取postid
def parse_one_page(self,response):
html = json.loads(response.text)
for one in html[‘Data‘][‘Posts‘]:
# postId ,two_url
postid = one[‘PostId‘]
url = self.two_url.format(postid)
yield scrapy.Request(
url=url,
callback=self.parse_two_page
)
# 解析二级页面
def parse_two_page(self,response):
item = TencentItem()
html = json.loads(response.text)
# 名称+类别+职责+要求+地址+时间
item[‘job_name‘] = html[‘Data‘][‘RecruitPostName‘]
item[‘job_type‘] = html[‘Data‘][‘CategoryName‘]
item[‘job_duty‘] = html[‘Data‘][‘Responsibility‘]
item[‘job_require‘] = html[‘Data‘][‘Requirement‘]
item[‘job_address‘] = html[‘Data‘][‘LocationName‘]
item[‘job_time‘] = html[‘Data‘][‘LastUpdateTime‘]
yield item
create database tencentdb charset utf8;
use tencentdb;
create table tencenttab(
job_name varchar(500),
job_type varchar(100),
job_duty varchar(2000),
job_require varchar(2000),
job_address varchar(100),
job_time varchar(100)
)charset=utf8;
管道文件pipelines.py实现
class TencentPipeline(object):
def process_item(self, item, spider):
print(dict(item))
return item
import pymysql
class TencentMysqlPipeline(object):
def open_spider(self,spider):
self.db = pymysql.connect(
‘localhost‘,‘root‘,‘123456‘,‘tencentdb‘,charset=‘utf8‘
)
self.cursor = self.db.cursor()
def process_item(self,item,spider):
ins=‘insert into tencenttab values(%s,%s,%s,%s,%s,%s)‘
L = [
item[‘job_name‘],
item[‘job_type‘],
item[‘job_duty‘],
item[‘job_require‘],
item[‘job_address‘],
item[‘job_time‘]
]
self.cursor.execute(ins,L)
self.db.commit()
return item
def close_spider(self,spider):
self.cursor.close()
self.db.close()
# 定义常用变量，添加管道即可
www.so.com -> 图片 -> 美女
2、F12抓包,抓取到json地址 和 查询参数(QueryString)
url = ‘https://image.so.com/zjl?ch=beauty&t1=595&src=banner_beauty&sn={}&listtype=new&temp=1‘
ch: beauty
t1: 595
src: banner_beauty
sn: 90
listtype: new
temp: 1
1、创建爬虫项目和爬虫文件
scrapy startproject So
cd So
scrapy genspider so image.so.com
2、定义要爬取的数据结构(items.py)
img_url = scrapy.Field()
img_title = scrapy.Field()
3、爬虫文件实现图片链接+名字抓取
import scrapy
import json
from ..items import SoItem
class SoSpider(scrapy.Spider):
name = ‘so‘
allowed_domains = [‘image.so.com‘]
# 重写start_requests()方法
url = ‘http://image.so.com/zjl?ch=beauty&sn={}&listtype=new&temp=1‘
def start_requests(self):
for sn in range(0,91,30):
full_url = self.url.format(sn)
# 扔给调度器入队列
yield scrapy.Request(
url=full_url,
callback=self.parse_image
)
def parse_image(self,response):
html = json.loads(response.text)
item = SoItem()
for img_dict in html[‘list‘]:
item[‘img_url‘] = img_dict[‘qhimg_url‘]
item[‘img_title‘] = img_dict[‘title‘]
yield item
4、管道文件（pipelines.py）
from scrapy.pipelines.images import ImagesPipeline
import scrapy
class SoPipeline(ImagesPipeline):
# 重写get_media_requests()方法
def get_media_requests(self, item, info):
yield scrapy.Request(
url=item[‘img_url‘],
meta={‘name‘:item[‘img_title‘]}
)
# 重写file_path()方法,自定义文件名
def file_path(self, request, response=None, info=None):
img_link = request.url
# request.meta属性
filename = request.meta[‘name‘] + ‘.‘ + img_link.split(‘.‘)[-1]
return filename
5、设置settings.py
IMAGES_STORE = ‘/home/tarena/images/‘
6、创建run.py运行爬虫
图片管道使用方法总结
1、爬虫文件: 将图片链接yield到管道
2、管道文件:
from scrapy.pipelines.images import ImagesPipeline
class XxxPipeline(ImagesPipeline):
def get_media_requests(self,xxx):
pass
def file_path(self,xxx):
pass
3、settings.py中:
IMAGES_STORE = ‘绝对路径‘
文件管道使用方法总结
1、爬虫文件: 将文件链接yield到管道
2、管道文件:
from scrapy.pipelines.images import FilesPipeline
class XxxPipeline(FilesPipeline):
def get_media_requests(self,xxx):
pass
def file_path(self,xxx):
return filename
3、settings.py中:
FILES_STORE = ‘绝对路径‘
字符串方法总结
1、strip()
2、split()
3、replace(‘‘,‘‘)
‘xx_xx_xx xx xx‘.replace(‘_‘,‘$‘).replace(‘ ‘,‘#‘)
4、‘‘.join()
5、字符串切片(正向切,反向切) : S[-10:]
scrapy.FormRequest(
url=posturl,
formdata=formdata,
callback=self.parse
)
1、创建项目+爬虫文件
scrapy startproject Youdao
cd Youdao
scrapy genspider youdao fanyi.youdao.com
2、items.py
result = scrapy.Field()
3、youdao.py
# -*- coding: utf-8 -*-
import scrapy
import time
import random
from hashlib import md5
import json
from ..items import YoudaoItem
class YoudaoSpider(scrapy.Spider):
name = ‘youdao‘
allowed_domains = [‘fanyi.youdao.com‘]
word = input(‘请输入要翻译的单词:‘)
def start_requests(self):
post_url = ‘http://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule‘
salt, sign, ts = self.get_salt_sign_ts(self.word)
formdata = {
‘i‘: self.word,
‘from‘: ‘AUTO‘,
‘to‘: ‘AUTO‘,
‘smartresult‘: ‘dict‘,
‘client‘: ‘fanyideskweb‘,
‘salt‘: salt,
‘sign‘: sign,
‘ts‘: ts,
‘bv‘: ‘cf156b581152bd0b259b90070b1120e6‘,
‘doctype‘: ‘json‘,
‘version‘: ‘2.1‘,
‘keyfrom‘: ‘fanyi.web‘,
‘action‘: ‘FY_BY_REALTlME‘
}
# 发送post请求的方法
yield scrapy.FormRequest(url=post_url,formdata=formdata)
def get_salt_sign_ts(self, word):
# salt
salt = str(int(time.time() * 1000)) + str(random.randint(0, 9))
# sign
string = "fanyideskweb" + word + salt + "n%A-rKaT5fb[Gy?;N5@Tj"
s = md5()
s.update(string.encode())
sign = s.hexdigest()
# ts
ts = str(int(time.time() * 1000))
return salt, sign, ts
def parse(self, response):
item = YoudaoItem()
html = json.loads(response.text)
item[‘result‘] = html[‘translateResult‘][0][0][‘tgt‘]
yield item
4、settings.py
1、ROBOTSTXT_OBEY = False
2、LOG_LEVEL = ‘WARNING‘
3、COOKIES_ENABLED = False
4、DEFAULT_REQUEST_HEADERS = {
"Cookie": "OUTFOX_SEARCH_USER_ID=970246104@10.169.0.83; OUTFOX_SEARCH_USER_ID_NCOO=570559528.1224236; _ntes_nnid=96bc13a2f5ce64962adfd6a278467214,1551873108952; JSESSIONID=aaae9i7plXPlKaJH_gkYw; td_cookie=18446744072941336803; SESSION_FROM_COOKIE=unknown; ___rl__test__cookies=1565689460872",
"Referer": "http://fanyi.youdao.com/",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
}
scrapy添加cookie的三种方式
# 1、修改 settings.py 文件
1、COOKIES_ENABLED = False -> 取消注释,开启cookie,检查headers中的cookie
2、DEFAULT_REQUEST_HEADERS = {} 添加Cookie
# 2、爬虫文件 - 利用cookies参数
COOKIES_ENABLED = True -> 修改为TRUE后，检查 Request()方法中cookies参数
def start_requests(self):
yield scrapy.Request(url=url,cookies={},callback=xxx)
# 3、DownloadMiddleware设置中间件
COOKIES_ENABLED = TRUE -> 找Request()方法中cookies参数
def process_request(self,request,spider):
request.cookies={}
1、调试蜘蛛的工具
2、交互式shell，可在不运行spider的前提下,快速调试 scrapy 代码(主要测试xpath表达式)
# scrapy shell URL地址
*1、request.url : 请求URL地址
*2、request.headers ：请求头(字典)
*3、request.meta ：item数据传递，定义代理(字典)
4、response.text ：字符串
5、response.body ：bytes
6、response.xpath(‘‘)
7、response.status : HTTP响应码
# 可用方法
shelp() : 帮助
fetch(request) : 从给定的请求中获取新的响应，并更新所有相关对象
view(response) : 在本地Web浏览器中打开给定的响应以进行检查
1、url
2、callback
3、headers
4、meta ：传递数据,定义代理
5、dont_filter ：是否忽略域组限制
默认False,检查allowed_domains[‘‘]
6、cookies
# settings.py
USER_AGENT = ‘‘
DEFAULT_REQUEST_HEADERS = {}
# spider
yield scrapy.Request(url,callback=函数名,headers={})
1、获取User-Agent
# 方法1 ：新建useragents.py,存放大量User-Agent，random模块随机切换
# 方法2 ：安装fake_useragent模块(sudo pip3 install fack_useragent)
from fake_useragent import UserAgent
ua_obj = UserAgent()
ua = ua_obj.random
2、middlewares.py新建中间件类
class RandomUseragentMiddleware(object):
def process_request(self,reuqest,spider):
ua = UserAgent()
request.headers[‘User-Agent‘] = ua.random
3、settings.py添加此下载器中间件
DOWNLOADER_MIDDLEWARES = {‘‘ : 优先级}
class RandomProxyDownloaderMiddleware(object):
def process_request(self,request,spider):
request.meta[‘proxy‘] = xxx
def process_exception(self,request,exception,spider):
return request
# 1、设置日志级别
LOG_LEVEL = ‘‘
# 2、保存到日志文件(不在终端输出)
LOG_FILE = ‘‘
# 3、设置数据导出编码(主要针对于json文件)
FEED_EXPORT_ENCODING = ‘‘
# 4、非结构化数据存储路径
IMAGES_STORE = ‘路径‘
FILES_STORE = ‘路径‘
# 5、设置User-Agent
USER_AGENT = ‘‘
# 6、设置最大并发数(默认为16)
CONCURRENT_REQUESTS = 32
# 7、下载延迟时间(每隔多长时间请求一个网页)
DOWNLOAD_DELAY = 3
# 8、请求头
DEFAULT_REQUEST_HEADERS = {}
# 9、添加项目管道
ITEM_PIPELINES = {}
# 10、添加下载器中间件
DOWNLOADER_MIDDLEWARES = {}
# 11、启用Cookie
COOKIES_ENABLED = True | False
1、spider
yield item[‘链接‘]
2、pipelines.py
from scrapy.pipelines.images import ImagesPipeline
import scrapy
class TestPipeline(ImagesPipeline):
def get_media_requests(self,item,info):
yield scrapy.Request(url=item[‘url‘],meta={‘name‘:item[‘name‘]})
def file_path(self,request,response=None,info=None):
name = request.meta[‘name‘]
filename = name
return filename
3、settings.py
IMAGES_STORE = ‘D:/Spider/images‘
# 参数
1、url
2、callback
3、headers
4、meta ：传递数据,定义代理
5、dont_filter ：是否忽略域组限制 - 默认False,检查allowed_domains[‘‘]
# request属性
1、request.url
2、request.headers
3、request.meta
4、request.method
# response属性
1、response.url
2、response.text
3、response.body
4、response.encoding
随机User-Agent（headers属性）
# 1、middlewares.py
class RandomUaDownloaderMiddleware(object):
def process_request(self,request,spider):
request.headers[‘User-Agent‘] = xxx
# 2、settings.py
DOWNLOADER_MIDDLEWARES = {‘xxx.middlewares.xxx‘:300}
随机代理（meta属性）
# 1、middlewares.py
class RandomProxyDownloaderMiddleware(object):
def process_request(self,request,spider):
request.meta[‘proxy‘] = xxx
def process_exception(self,request,exception,spider):
return request
# 2、settings.py
DOWNLOADER_MIDDLEWARES = {‘xxx.middlewares.xxx‘:200}
设置Cookie
# 1、middlewares.py
class RandomProxyDownloaderMiddleware(object):
def process_request(self,request,spider):
"""此处需要把cookies处理为字典"""
request.cookies = {}
def process_exception(self,request,exception,spider):
return request
# 2、settings.py
DOWNLOADER_MIDDLEWARES = {‘xxx.middlewares.xxx‘:200}
方法
scrapy.FormRequest(url=url,formdata=formdata,callback=self.xxx)
使用cookie
【1】方法1
COOKIES_ENABLED = False
DEFAULT_REQUEST_HEADERS = {‘Cookie‘:‘xxxx‘}
【2】方法2
COOKIES_ENABLED = True
yield scrapy.Request(url=url,cookies={},callback=self.xxxx)
yield scrapy.Request(url=url,formdata={},cookies={},callback=self.xxxx)
【3】方法3
COOKIES_ENBALED = True
class XxxCookieDownloaderMiddleware(object):
def process_request(self,request,spider):
request.cookies = {}
原理
多台主机共享1个爬取队列
实现
重写scrapy调度器(scrapy_redis模块)
sudo pip3 install scrapy_redis
为什么使用redis
1、Redis基于内存,速度快
2、Redis非关系型数据库,Redis中集合,存储每个request的指纹
3、scrapy_redis安装
sudo pip3 install scrapy_redis
GitHub地址
https://github.com/rmax/scrapy-redis
settings.py说明
# 重新指定调度器: 启用Redis调度存储请求队列
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
# 重新指定去重机制: 确保所有的爬虫通过Redis去重
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
# 不清除Redis队列: 暂停/恢复/断点续爬(默认清除为False,设置为True不清除)
SCHEDULER_PERSIST = True
# 优先级队列 （默认）
SCHEDULER_QUEUE_CLASS = ‘scrapy_redis.queue.PriorityQueue‘
#可选用的其它队列
# 先进先出
SCHEDULER_QUEUE_CLASS = ‘scrapy_redis.queue.FifoQueue‘
# 后进先出
SCHEDULER_QUEUE_CLASS = ‘scrapy_redis.queue.LifoQueue‘
# redis管道
ITEM_PIPELINES = {
‘scrapy_redis.pipelines.RedisPipeline‘: 300
}
#指定连接到redis时使用的端口和地址
REDIS_HOST = ‘localhost‘
REDIS_PORT = 6379
1、settings.py
# 1、使用scrapy_redis的调度器
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
# 2、使用scrapy_redis的去重机制
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
# 3、是否清除请求指纹,True:不清除 False:清除（默认）
SCHEDULER_PERSIST = True
# 4、(非必须)在ITEM_PIPELINES中添加redis管道
‘scrapy_redis.pipelines.RedisPipeline‘: 200
# 5、定义redis主机地址和端口号
REDIS_HOST = ‘127.0.0.1‘
REDIS_PORT = 6379
修改管道
ITEM_PIPELINES = {
‘Tencent.pipelines.TencentPipeline‘: 300,
# ‘scrapy_redis.pipelines.RedisPipeline‘: 200
‘Tencent.pipelines.TencentMysqlPipeline‘:200,
}
代码拷贝一份到分布式中其他机器，两台或多台机器同时执行此代码
使用redis_key改写
# 第一步: settings.py无须改动
settings.py和上面分布式代码一致
# 第二步:tencent.py
from scrapy_redis.spiders import RedisSpider
class TencentSpider(RedisSpider):
# 1. 去掉start_urls
# 2. 定义redis_key
redis_key = ‘tencent:spider‘
def parse(self,response):
pass
# 第三步:把代码复制到所有爬虫服务器，并启动项目
# 第四步
到redis命令行，执行LPUSH命令压入第一个要爬取的URL地址
>LPUSH tencent:spider 第1页的URL地址
# 项目爬取结束后无法退出，如何退出？
setting.py
CLOSESPIDER_TIMEOUT = 3600
# 到指定时间(3600秒)时,会自动结束并退出
原文：https://www.cnblogs.com/exlo/p/14160005.html