3 kinds of bots
- Training bots collect text to train models, for example GPTBot from OpenAI and ClaudeBot from Anthropic.
- Search bots build the index that assistants search, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- User bots open a page because a person asked for it, for example ChatGPT-User, Claude-User and Perplexity-User. Because a person triggered the visit, some of them do not follow robots.txt.
Google and Apple work differently
- Google-Extended is not a crawler but a word in robots.txt. It decides whether Google may use your content to train Gemini and to ground answers in the Gemini apps. Google states that it does not affect Google Search or your ranking.
- Applebot-Extended lets you decide whether Apple may use your content to train its AI models.
An example
This robots.txt keeps OpenAI’s training bot out, while ChatGPT search can still show your pages:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /Our recommendation
Allow the search bots, otherwise you cannot appear in AI answers. Decide about the training bots: if your texts are your product, like a publisher’s, blocking can make sense. For most service companies it is an advantage when AI models know who they are and what they offer. Our own website lets all of them in.
Check what you have
Open yourdomain.ch/robots.txt in your browser. Some hosting providers and security services block AI bots by default, sometimes without telling you. More on the bigger picture in how AI assistants find websites.
Questions and answers
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you can allow separately.
Does Google-Extended affect my Google ranking?
No. Google states that Google-Extended is not a ranking signal and does not affect inclusion in Google Search.