GPTBot, Google-Extended and co.: should you block AI crawlers?

Short answer: For most small businesses that want customers to find them, the sensible default is to allow the search and user-fetch bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the like) and decide separately whether to allow the training bots (GPTBot, ClaudeBot, Google-Extended). The vendors now split these roles into different user agents, so blocking training no longer has to mean disappearing from AI search. Blocking Google-Extended doesn't affect your Google Search ranking, according to Google.

The mistake to avoid is a blanket "block all AI" rule copied from a forum. It can quietly remove you from the answers your customers are starting to read.

Three kinds of AI bot

Each vendor's own documentation (as of November 2026) separates its bots by job:

Training crawlers collect content that may be used to train AI models.
Search crawlers index pages so they can appear, with links, in the vendor's AI search answers.
User-triggered fetchers visit a page because a person asked the assistant something right now.

OpenAI

GPTBot: crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals your content shouldn't be used for training.
OAI-SearchBot: surfaces websites in ChatGPT's search features. OpenAI says sites that opt out won't be shown in ChatGPT search answers, though they can still appear as navigational links, and that a robots.txt change can take about 24 hours to take effect for search.
ChatGPT-User: visits pages when a user asks ChatGPT or a custom GPT something. OpenAI says it isn't used for automatic crawling, robots.txt rules may not apply because the actions are user-initiated, and it isn't used to decide search inclusion.

OpenAI also states each setting is independent: you can allow OAI-SearchBot and disallow GPTBot.

Google

Google-Extended: not a separate crawler but a robots.txt token. Google says it controls whether content it crawls may be used to train future Gemini models and for grounding in Gemini Apps and on Vertex AI, and that it doesn't affect inclusion in Google Search and isn't a ranking signal.
Googlebot: your normal Search crawler. Never block it unless you want to leave Google Search.

Google's crawler documentation also notes it intends to support a shared machine-readable training preference signal in early 2027, so this area is still moving.

Anthropic

ClaudeBot: collects web content that could contribute to model training. Restricting it signals future materials should be excluded from training.
Claude-SearchBot: indexes content to improve search results. Anthropic says disabling it may reduce your visibility in user search results.
Claude-User: fetches pages when a user asks Claude a question. Disabling it stops Claude retrieving your content for those questions.

Anthropic adds that IP blocking may not work reliably as an opt-out, because it stops their bots reading your robots.txt.

Perplexity

PerplexityBot: surfaces and links websites in Perplexity's search results; Perplexity says it isn't used to crawl content for AI foundation models.
Perplexity-User: fetches pages for user questions; Perplexity says it generally ignores robots.txt because the fetch is user-requested.

Decision matrix

Your situation	Training bots (GPTBot, ClaudeBot, Google-Extended)	Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot)	User fetchers (ChatGPT-User, Claude-User, Perplexity-User)
Local service business wanting inquiries	Your choice; little downside either way	Allow	Allow
Online shop	Your choice	Allow	Allow
Publisher or course creator whose content is the product	Consider blocking	Allow if you want referral traffic	Allow, or restrict paid sections by login
Members-only or paid content	Block	Block the gated paths	Gate with login, not robots.txt
Staging or internal tools	Block everything	Block	Protect with a password

robots.txt variants

Variant A: Allow everything (the default if you do nothing)

No AI-specific rules needed. If your robots.txt has no group for a bot, it falls back to the User-agent: * rules.

Variant B: Stay in AI search, opt out of training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

The explicit Allow groups aren't strictly required, but they make your intent obvious to whoever edits the file next.

Variant C: Keep paid or private sections out of everything

User-agent: *
Disallow: /members/
Disallow: /checkout/

Remember Google's own warning: robots.txt can't enforce crawler behavior, and well-behaved crawlers obey it while others might not. Anything genuinely private needs a login.

Practical checks

One file per host. Anthropic's page reminds you to update robots.txt on every subdomain you want covered, such as shop.example.com.
Check your firewall or CDN. Bot protection can block search bots you meant to allow. Perplexity's docs, for example, describe allowing its bots in a web application firewall using its published IP ranges.
Look at your logs. User-agent strings and published IP lists let you confirm which bots actually visit.
Revisit twice a year. Vendors add and rename bots; each keeps its list on the pages cited here.

FAQ

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search, according to OpenAI, as long as OAI-SearchBot is allowed. The two settings are independent.

Does blocking Google-Extended hurt my Google ranking?

Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal.

Can I block AI Overviews with Google-Extended?

Google describes Google-Extended in terms of Gemini training and grounding in Gemini Apps and Vertex AI. AI features in Google Search are governed by Search's own controls; check Google's current documentation before relying on any token for that.

Is robots.txt enough to protect my content?

No. It's a request that reputable bots honor. Use logins or paywalls for anything you truly need to protect.

Before you block anything, a second pair of eyes on robots.txt and your firewall rules is cheap insurance: send us your domain.