scrapy startproject 项目名
scrapy crawl XX
scrapy crawl quotes -o quotes.json
scrapy crawl quotes -o quotes.jl
scrapy shell http://www.scrapyd.cn
scrapy genspider example example.com#创建蜘蛛，蜘蛛名为example
.extract_first() .extract() .get() .getall()
.intro #class = "intro"
#firstname #id = "firstname"
标签名::attr(属性名) #“a::attr(href)” "img::attr(src)"
标签名::text #"a::text" “a *::text”#a标签的所有文字
div,p#选择<div>元素内的所有<p>元素 div p#选择<div>元素内的所有<p>元素
div>p#选择所有父级是 <div> 元素的 <p> 元素 div+p#选择所有紧接着<div>元素之后的<p>元素
[target]#选择所有带有target属性元素,[target=blank],[target~=blank],[target|=blank]
string()#文本整段提取（拼接）
/#从根节点选取 //#从匹配选择的当前节点选择文档中的节点，而不考虑它们的位置。
@#选取属性“//@href” 标签[@属性名=属性值]
//text()#标签文本内容
https://docs.python.org/3/library/re.html
import scrapy class AuthorSpider(scrapy.Spider): name = ‘author‘ start_urls = [‘http://quotes.toscrape.com/‘] def parse(self, response): # follow links to author pages for href in response.css(‘.author + a::attr(href)‘): yield response.follow(href, self.parse_author) # follow pagination links for href in response.css(‘li.next a::attr(href)‘): yield response.follow(href, self.parse) def parse_author(self, response): def extract_with_css(query): return response.css(query).get(default=‘‘).strip() yield { ‘name‘: extract_with_css(‘h3.author-title::text‘), ‘birthdate‘: extract_with_css(‘.author-born-date::text‘), ‘bio‘: extract_with_css(‘.author-description::text‘), }
原文：https://www.cnblogs.com/puddingsmall/p/12242183.html