User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
User-agent: coccocbot-web/1.0
Disallow: /
User-agent: linkdexbot/2.0
Disallow: /
User-agent: DotBot/1.1
Disallow: /
User-Agent: YisouSpider
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: BOT
Disallow: /
User-Agent: CrawlDaddy
Disallow: /
User-Agent: ApacheBench
Disallow: /
User-Agent: Swiftbot
Disallow: /
User-Agent: AhrefsBot
Disallow: /
User-Agent: ZmEu
Disallow: /
User-Agent: WinHttp
Disallow: /
User-Agent: EasouSpider
Disallow: /
User-Agent: HttpClient
Disallow: /
User-Agent: YYSpider
Disallow: /
User-Agent: jaunty
Disallow: /
User-Agent: oBot
Disallow: /
User-Agent: Linguee Bot
Disallow: /
User-Agent: Bytespider
Disallow: /
User-Agent: BLEXBot
Disallow: /
User-Agent: CompSpyBot
Disallow: /
User-Agent: Exabot
Disallow: /
User-Agent: ZoominfoBot
Disallow: /
User-Agent: ExtLinksBot
Disallow: /
User-Agent: AlphaBot
Disallow: /
User-Agent: perl
Disallow: /
User-Agent: Wget
Disallow: /
User-Agent: ZmEu
Disallow: /
User-Agent: Python
Disallow: /
User-Agent: mail.RU
Disallow: /
User-Agent: ApacheBench
Disallow: /
User-Agent: Swiftbot
Disallow: /
User-Agent: AhrefsBot
Disallow: /
User-Agent: ZmEu
Disallow: /
User-Agent: WinHttp
Disallow: /
User-Agent: EasouSpider
Disallow: /
User-Agent: HttpClient
Disallow: /
User-Agent: YYSpider
Disallow: /
User-Agent: jaunty
Disallow: /
User-Agent: oBot
Disallow: /
User-Agent: Linguee Bot
Disallow: /
User-Agent: Bytespider
Disallow: /
User-Agent: BLEXBot
Disallow: /
User-Agent: CompSpyBot
Disallow: /
User-Agent: Exabot
Disallow: /
User-Agent: ExtLinksBot
Disallow: /
User-Agent: AlphaBot
Disallow: /
User-Agent: perl
Disallow: /
User-Agent: Wget
Disallow: /
User-Agent: ZmEu
Disallow: /
User-Agent: Python
Disallow: /
User-Agent: mail.RU
Disallow: /
User-Agent: Go-http-client
Disallow: /
User-agent: *
Disallow: /admin/
Disallow: /adminlogin/
Disallow: /log/
Disallow: /update/
Disallow: /history/
Disallow: /test/
Disallow: /data/
都是一些无效的爬虫访问
网站迁移服务器后CPU、内存飙升，设置robots.txt 问题
原文：https://www.cnblogs.com/zhian/p/12967875.html