robots.txt is a public text file at yourdomain/robots.txt. It tells crawlers what they may read. It is a voluntary standard (RFC 9309), not a lock.
Each block starts with “User-agent:” (the bot's name) and then “Disallow:” or “Allow:” rules. A bot that is not named follows the “User-agent: *” block.
Add a “Sitemap:” line so crawlers find all your pages.
Do it on your siteOpen yourdomain/robots.txt. Check that Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot are not blocked, and that a Sitemap line exists.
A bot is not named anywhere in robots.txt. Which rules apply to it?
Show the answer
Answer: The “User-agent: *” block. Bots that are not named follow the rules for all user agents (*).
Training bots collect text for future models: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
Search bots index pages for AI answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking Claude-SearchBot may reduce your visibility in Claude's search results, says Anthropic.
User bots fetch a page when a person asks: ChatGPT-User, Claude-User, Perplexity-User. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity-User generally ignores it; Anthropic says Claude-User respects it.
Do it on your sitePlay the bot game in this Academy until you score 8/8.
Which of these is a training bot, not a search bot?
Show the answer
Answer: GPTBot. OpenAI: GPTBot collects content that may be used to train its foundation models.
You can say no to training and still be found. Name the training bots with “Disallow: /” and the search bots with “Allow: /”.
Blocking Google-Extended does not remove you from AI Overviews: those are controlled through Googlebot and snippet rules.
Whether to block training is your decision. There is no measured evidence either way for small organisations: record it as a conscious choice.
Do it on your siteCopy the template from the Agent's fix pack (or this lesson's source), adapt the sitemap URL, and ask whoever runs your site to publish it.
You block Google-Extended. What happens to your AI Overviews?
Show the answer
Answer: Nothing changes: they depend on Googlebot. Google: Google-Extended covers Gemini training and grounding in some other systems, not Search's AI features.
A perfect robots.txt is useless if your hosting or CDN blocks the bot first. Google lists checking “any CDN or hosting infrastructure” as a best practice.
Since 1 July 2025, Cloudflare blocks AI crawlers by default on new domains and offers Pay Per Crawl.
Since 15 September 2026, Cloudflare's “Block” and “Block on pages with ads” settings also stop mixed-use crawlers, including Googlebot, Bingbot and Applebot, so they cut you out of search too. To refuse training but stay in search, use the new “Disallow AI Training” setting.
Do it on your siteAsk your web host or check your Cloudflare dashboard: which AI crawlers are blocked? Allow the search bots you want.
robots.txt allows everything. Can AI search bots still be blocked?
Show the answer
Answer: Yes, by the CDN or firewall. Security layers can block bots before they ever read robots.txt.
To limit what Google shows from a page, use nosnippet, data-nosnippet (for parts of a page), max-snippet or noindex.
Since 31 August 2026, every site has a Search Console setting called “Search generative AI”: you can exclude your site from AI Overviews, AI Mode and AI features in Discover.
It does not cover the Gemini app or AI training: that is still Google-Extended. Sites that opt out get no traffic or impressions from these features; Google says it is not a ranking signal elsewhere.
Do it on your siteIf one paragraph should never be quoted (for example outdated prices), wrap it in data-nosnippet instead of hiding the whole page.
What does Google's 2026 AI opt-out cost you?
Show the answer
Answer: All traffic and impressions from its AI features. Google: sites that opt out will not receive traffic or impressions from its generative AI features.