5 Best Ways to Fix robots.txt for AI Crawlers, After Checking 219 Domains

Intro

Every article about AI crawlers warns you about the same mistake. You blocked GPTBot during the 2023 scraping panic, the warning goes, and you accidentally caught the newer agents that fetch pages to cite in live answers. You meant to stop the training scrapers and you quietly opted out of being recommended.

We checked whether that is actually happening.

Across 219 domains, it happened zero times.

Not rarely. Not in a handful of cases. Not once.

Method

We took the top twenty organic results for 26 commercial queries across seven categories: SaaS, ecommerce, finance, health, travel, home and marketing. That produced 254 unique domains. We fetched robots.txt for each on 28 July 2026, and 219 resolved. Seven of those served no robots.txt at all, which counts as allowing everything.

For each domain we resolved every AI user agent against the file using the standard precedence rule: the most specific matching User-agent group wins, so an exact agent match overrides *. A domain counts as blocking an agent when its winning group disallows the site root without a more specific allow.

Twelve agents were checked, split by what they feed.

Agent	Operator	Feeds
GPTBot	OpenAI	training
OAI-SearchBot	OpenAI	ChatGPT search citations
ClaudeBot	Anthropic	training
Claude-SearchBot	Anthropic	Claude search citations
PerplexityBot	Perplexity	answers
DuckAssistBot	DuckDuckGo	answers
Amazonbot	Amazon	answers
CCBot	Common Crawl	a corpus many models draw on
Google-Extended	Google	Gemini training, not Search
Applebot-Extended	Apple	training
Bytespider	ByteDance	training
meta-externalagent	Meta	training

Results

77% of these domains allow every AI agent. Twenty-three percent block at least one.

Agent	Blocked	Domains naming it explicitly
Bytespider	20%	45
CCBot	20%	47
ClaudeBot	17%	50
Google-Extended	16%	46
meta-externalagent	16%	38
PerplexityBot	15%	44
GPTBot	14%	53
Applebot-Extended	14%	39
Amazonbot	13%	39
Claude-SearchBot	10%	22
DuckAssistBot	10%	24
OAI-SearchBot	9%	34

The agent blocked least often is the one that fetches pages for ChatGPT to cite. The agents blocked most often are the ones that scrape for training corpora.

That ordering is not an accident, and it is the whole finding.

Nobody is making the mistake. Everybody is making a decision

Look at the two operators who run both a training crawler and a search crawler.

	Blocks both	Blocks training, allows search	Blocks search, allows training
OpenAI, GPTBot / OAI-SearchBot	9%	5%	0%
Anthropic, ClaudeBot / Claude-SearchBot	10%	7%	0%

Twelve domains block GPTBot while deliberately allowing OAI-SearchBot. Fifteen do the equivalent for Anthropic. That is a coherent policy: do not train on my content, but do cite me.

Nobody does the reverse.

Two more numbers point the same way. Only four domains in the entire sample use a blanket User-agent: * root disallow, and exactly one blocks an AI agent without naming any AI agent explicitly. Whoever edits these files knows which agents exist and what each one does.

So the industry warning addresses a problem that, in this sample, does not exist. The interesting question is not whether you blocked the search agents by accident. It is whether you meant to.

Because 18% of these domains block at least one search agent and 4% block all four. Those sites have opted out of appearing in AI answers, and on this evidence they did it on purpose.

The 5 ways to fix yours today

1. Treat training and search as two separate questions. They are different decisions and the market has already split them. Most publishers want the second without the first. Write the file that way rather than treating "AI crawlers" as one category.

2. Name every agent explicitly. Only four domains here rely on the wildcard, and for good reason: a bare User-agent: * disallow catches agents that did not exist when you wrote it, and will catch agents that do not exist yet. Two of the most-blocked agents in these results, Bytespider and meta-externalagent, postdate most of the robots.txt advice still circulating.

3. Check OAI-SearchBot and Claude-SearchBot by name. These two decide whether you can be cited. Nine and ten percent of the sample block them. If you are in that group and did not intend to be, this is the line to change, and it is the only item on this list with an immediate visibility cost.

4. Re-audit quarterly. New agents appeared several times in the past year. A file written eighteen months ago is missing entries no matter how carefully it was written at the time.

5. Verify at the edge, not only in the file. Bot-management rules at your CDN can block an agent that robots.txt explicitly permits, and the file will tell you nothing about it. This is the failure that survives a perfect robots.txt.

A starting point that separates the two decisions:

# Cite me. These fetch pages to quote in live answers.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: DuckAssistBot
Allow: /

# Do not train on me. Flip these to Allow if you disagree.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

That is close to the configuration twelve to fifteen domains in this sample already run. It is a mainstream position now rather than a clever one.

Why none of this shows up in a report you already run

Search Console reports Googlebot. It says nothing about OAI-SearchBot. Analytics reports sessions, and an engine that never cites you sends no sessions to be missing. Rank trackers report positions, and your positions are unaffected by any of this.

There are two ways to catch it. Read the file directly, which is what this study did. Or work backwards from the symptom with an AI visibility checker, asking the engines your category questions on a schedule and watching for the answer that says they cannot see you at all.

The crawler audit behind this study is the one we run in Honeyb, resolving each named agent in the catalogue above against a site's robots rules on a schedule rather than as a one-off. Access is the first of four things it checks. The other three are whether the engines recommend you, what specifically to change, and then producing and shipping the content that changes it.

Two limits worth stating

This sample is US commercial SERPs, so it skews towards large publishers and established brands, who have both the awareness and the legal advice to make a considered decision here. A sample of small business sites would very likely look different, and the accidental-blocking story might well turn out to be true there. We would run that next.

And a robots.txt file records intent, not enforcement. We did not test whether these domains actually serve content to each agent, which is where CDN rules live and where the gap between the file and reality tends to open.

The platform behind this study is at honeyb.ai.