AI robots.txt generator
Block AI training, SEO tools or any single bot, and keep search and AI answers.
Search engines 20
Build search indexes. Google's and Bing's also feed their AI answers.
AI search 21
Build the indexes AI search engines and search APIs answer from.
Live fetch 22
Open a page when someone asks an assistant or agent to.
Training 16
Collect pages to train models.
SEO and AI visibility tools 11
Third-party crawlers for SEO and AI visibility data.
Ads checks 3
Check landing pages for ad platforms.
# Training User-agent: Applebot-Extended User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: Bytespider User-agent: CCBot User-agent: cohere-training-data-crawler User-agent: DeepSeekBot User-agent: Qwenbot User-agent: ChatGLM-Spider User-agent: KimiBot User-agent: ElevenlabsCrawler User-agent: AI2Bot Disallow: / User-agent: * Allow: /
What each bot does: AI crawler and bot list