Someone adds Disallow: / for every agent with “GPT” or “Claude” in the name, and support tickets
start arriving about assistants that cannot read the quickstart. The blanket rule caught the
fetchers that act for a developer mid-task, which is the traffic you wanted. The training crawlers
it was aimed at were a different set of names.
What you get
You will end up with a robots.txt that refuses bulk collection while leaving your documentation readable by an agent working for a person. You also get a matcher you can run the rules through before deploying them. This is for you if you publish developer documentation.
Short answer
Write one group for the training crawlers and a separate group for the fetchers that act for a user
in the moment, then keep a wildcard group for everything unnamed. GPTBot, ClaudeBot, CCBot,
Google-Extended, and Applebot-Extended belong in the first. ChatGPT-User, Claude-User, and
PerplexityBot belong in the second, with your docs allowed.
You will need
A site whose robots.txt you control, and Node 22 or later to run the matcher. The rules follow RFC 9309, which standardized the protocol that had been a convention since 1994.1 Take the agent names from the operators rather than from a blog post. Each of them publishes a crawler page, and each page changes: see the OpenAI list, the Anthropic list, and the Google list.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| A noindex meta tag | You want a page fetched but kept out of a search index | The page has to be fetched for the tag to be read, so collection still happens | The goal is to stop the fetch rather than the listing |
| Edge blocking on user agent | A crawler ignores the file and you need the refusal enforced | Rules that can catch a browser sharing a token, and a support load when they do | The crawler respects robots.txt and no enforcement is needed |
| No rules at all | A small site with nothing you would mind seeing in a model | No control over bulk collection, and no signal about what you intended | You have private paths, or a view on training use |
| robots.txt groups | Any public site, as the statement of intent every other layer refers back to | An honor system, so a crawler that ignores it is unaffected | The traffic is hostile rather than merely unwanted |
Two of these are requests and two are enforcement. robots.txt and a meta tag say what you want, and a crawler that ignores them faces nothing. Edge rules actually stop traffic, and they cost you false positives: an agent that sets a browser user agent gets through, and a developer using a scripted client sometimes does not. Run both, and treat the file as the statement of record.
Split the named agents into two groups
The split is between collection for training and retrieval for a person who is waiting.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /docs/
Allow: /openapi.json
Disallow: /account/
Disallow: /admin/
User-agent: *
Disallow: /account/
Disallow: /admin/Google-Extended is the one most often misread.2 It governs whether your content feeds Gemini and
the models behind it, and it has no effect on Googlebot’s search crawling. Adding it does not
remove you from search results, and removing it does not put you back in.
The wildcard group is the one an unnamed crawler lands in, and a new agent name appears every few months. Keep that group permissive enough to stay useful and strict about the paths that are private, because it is the rule that governs every crawler you have not heard of.
Order inside a group does not matter, and order between groups does not either. That surprises people who write the file expecting the first match to win. Specificity decides the group, and path length decides the rule, so moving lines around changes nothing.3
Run the rules before you deploy them
Rule matching has two parts people get wrong: which group applies to an agent, and which rule inside that group wins.
export function isAllowed(text, agent, path) {
const group = groupFor(parse(text), agent)
let winner = { allow: true, path: '' }
for (const rule of group.rules) {
if (!path.startsWith(rule.path)) continue
const longer = rule.path.length > winner.path.length
const tie = rule.path.length === winner.path.length && rule.allow
if (longer || tie) winner = rule
}
return winner.allow
}One group applies, not all of them. The most specific matching user-agent line wins, and the
wildcard group is ignored entirely for an agent named anywhere else in the file. A team that adds a
name to the strict group and expects the wildcard Disallow rules to still apply has opened the
private paths to that one crawler.
Check it worked
Run every agent against every path that matters and read the grid.
node demo.mjsagent /docs/quickstart /openapi.json /account/billing /blog/post
GPTBot disallow disallow disallow disallow
ClaudeBot disallow disallow disallow disallow
ChatGPT-User allow allow disallow allow
Claude-User allow allow disallow allow
PerplexityBot allow allow disallow allow
Googlebot allow allow disallow allowThe first two rows are the collection crawlers, refused everywhere. The next three are the fetchers acting for a person, which keep the documentation and the OpenAPI description while losing the account area. The last row is search, unaffected by any of it, which is the outcome the blanket rule would have destroyed.
node --test match.test.mjs1..5
# tests 5
# suites 0
# pass 5
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 110.650071When it goes wrong
An assistant reports it cannot read your docs. The agent matched a group you meant for training crawlers, because its name contains a string you matched loosely. Match the full token, and test the exact name from your access logs.
Search traffic drops after a robots change. A crawler was refused a path that search depends on, or
a Disallow prefix matched more paths than anyone intended. Compare before and
after with the matcher, path by path.
The rules have no effect at all. The file is served from the wrong host or with an HTML media type. It must sit at the root of the exact scheme, host, and port, and be served as plain text.
When not to do this
Do not use robots.txt to hide anything. The file is public, and a list of paths you would rather nobody fetched is a map for anyone curious. Put authentication in front of private routes, and leave the file for crawler policy.
Do not refuse the user-triggered fetchers to make a point about training. A developer asking an assistant how to call your API is the case your documentation exists for, and refusing it sends them to a competitor whose docs load. Keep that lane open.
Do not copy a rule set from another site without reading it. Names go out of date, one agent split into two, and a stale list refuses a crawler that no longer exists while missing the one that replaced it. Re-read the operators’ own pages before each edit to the file.
Do not treat the file as a legal position. It records an intention, and it binds nobody who chooses to ignore it. If the use of your content matters commercially, the terms of service and the edge rules are where that argument is actually made.
Related how-tos
Last verified
Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.
Footnotes
-
RFC 9309 is dated September 2022, and its first author is M. Koster, who wrote the original as a consensus of the robots mailing list on June 30, 1994. Twenty-eight years is a long apprenticeship for a text file. The RFC adds definition language, error handling and caching to the 1994 method, and sets the floor on any parsing limit a crawler imposes at 500 KiB. ↩︎ Back to text
-
Google’s crawler page states that
Google-Extendedhas no HTTP user agent string of its own. Crawling is done under the existing Google user agents, and the token is, in the page’s words, used in a control capacity. Nothing on the wire ever announces itself by that name, so an access log never shows it. The example group the page gives allows/archive/1Q84and disallows the rest of/archive/, which is the most specific match rule in two lines. ↩︎ Back to text -
Length in RFC 9309 is counted in octets, the word the RFC uses, rather than in characters or lines. The most specific match must be used, and where an
allowand adisalloware equivalent theallowshould win, which is why the matcher on this page breaks ties that way. The same section adds that duplicate rules in a group may be deduplicated, and that a group with no rules allows everything. ↩︎ Back to text