Does blocking GPTBot keep my site out of ChatGPT?

GPTBot only collects training data. ChatGPT search uses OAI-SearchBot and live answers use ChatGPT-User, each with its own robots.txt token. A GPTBot rule touches neither.

Can robots.txt stop ChatGPT-User from opening my pages?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the page. Perplexity-User and Meta-ExternalFetcher work the same way.

Does blocking Google-Extended hurt me in Google Search?

Google says Google-Extended doesn't affect inclusion or ranking in Search. It controls Gemini training and Gemini grounding, not AI Overviews or AI Mode.

How do I know a request from a bot is real?

Check it against the operator's published IP ranges or reverse DNS. Each bot in the list says which method its operator offers.

AI crawler and bot list

I built this list to sort and monitor the bots showing up in my clients' server logs. There are thousands more, but these are the ones that matter for SEO and AEO.

Generate a robots.txt GitHub

99 bots

Operator: Google
robots.txt: User-agent: Googlebot-ImageFollows robots.txt
User agent: Googlebot-Image/1.0
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: Googlebot-VideoFollows robots.txt
User agent: Googlebot-Video/1.0
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: Googlebot-NewsFollows robots.txt
User agent: Plain browser, can't be told apart
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: GoogleOtherFollows robots.txt
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/131.0.0.0 Safari/537.36
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: GooglebotFollows robots.txt
User agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: Google-InspectionToolFollows robots.txt
User agent: Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)
Verify: Published IP ranges
Docs: developers.google.com

Operator: Google
robots.txt: User-agent: Storebot-GoogleFollows robots.txt
User agent: Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
Verify: Published IP ranges
Docs: developers.google.com

Operator: Microsoft
robots.txt: User-agent: BingbotFollows robots.txt
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Verify: Published IP ranges
Docs: bing.com

Operator: Apple
robots.txt: User-agent: ApplebotFollows robots.txt
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Verify: Published IP ranges
Docs: support.apple.com

Operator: DuckDuckGo
robots.txt: User-agent: DuckDuckBotFollows robots.txt
User agent: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Verify: Published IP ranges
Docs: duckduckgo.com

Operator: Yandex
robots.txt: User-agent: YandexBotFollows robots.txt
User agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Verify: Reverse DNS
Docs: yandex.com

Operator: Yahoo
robots.txt: User-agent: SlurpFollows robots.txt
User agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Verify: No published method
Docs: help.yahoo.com

Operator: Baidu
robots.txt: User-agent: BaiduspiderFollows robots.txt
User agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Verify: Reverse DNS
Docs: help.baidu.com

Operator: Naver
robots.txt: User-agent: YetiFollows robots.txt
User agent: Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)
Verify: Reverse DNS
Docs: searchadvisor.naver.com

Operator: Seznam
robots.txt: User-agent: SeznamBotFollows robots.txt
User agent: Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)
Verify: Reverse DNS
Docs: o-seznam.cz

Operator: Huawei
robots.txt: User-agent: PetalBotFollows robots.txt
User agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Verify: Reverse DNS
Docs: webmaster.petalsearch.com

Operator: Mojeek
robots.txt: User-agent: MojeekBotFollows robots.txt
User agent: Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html)
Verify: No published method
Docs: mojeek.com

Operator: Qwant
robots.txt: User-agent: QwantbotFollows robots.txt
User agent: Mozilla/5.0 (compatible; Qwantbot/1.0; +https://help.qwant.com/bot/)
Verify: No published method
Docs: help.qwant.com

Operator: LY Corporation (Yahoo Japan)
robots.txt: User-agent: Y!JNot documented
User agent: Y!J-BRW/1.0 (https://support.yahoo-net.jp/PccSearch/s/article/H000007955)
Verify: No published method
Docs: support.yahoo-net.jp

Operator: Coc Coc
robots.txt: User-agent: coccocbotFollows robots.txt
User agent: Mozilla/5.0 (compatible; coccocbot-web/1.1; +http://help.coccoc.com/searchengine)
Verify: No published method
Docs: help.coccoc.com

FAQ

Frequently asked questions

Block only the training crawlers. Search engines, AI search and live fetches still reach every page. Paste the file into robots.txt above your other rules. opt-out-of-training.txt