中国最后一个老实人 常见垃圾蜘蛛收集 – robots.txt – 文件，国外垃圾蜘蛛分类大全，网站卡顿解决办法
在根目录创建一个robots.txt文件，再复制以下代码进去。
#SEO/数据分析类
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: netpeakbot
Disallow: /
User-agent: proximic
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: sistrix
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Spinn3r
Disallow: /
User-agent: SerpstatBot
Disallow: /
#数据聚合/商业类
User-agent: Amazonbot
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: Barkrowler
Disallow: /
User-agent: LinkpadBot
Disallow: /
User-agent: SiteExplorerBot
Disallow: /
User-agent: DataMiner
Disallow: /
User-agent: Leadfeeder
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: AwarioBot
Disallow: /
#其他高频爬虫
User-agent: MojeekBot
Disallow: /
User-agent: AddThis Bot
Disallow: /
User-agent: webmeup
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: hubspot
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: CrawlMonster
Disallow: /
User-agent: GDELTBot
Disallow: /
User-agent: FacebookExternalHit
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: MauiBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: babbar
Disallow: /
User-agent: BingSapphire
Disallow: /
