# How to set robots.txt rules for each AI crawler > Separate the crawlers that train models from the fetchers acting for a person right now, and check each rule against the matching algorithm before you ship it. Source: https://voxgig.com/howto/set-robots-rules-for-ai-crawlers - Audience: platform-team - Level: beginner - Verified: 2026-09-14 - Published: 2026-09-14 - Updated: 2026-09-24 ## Short answer Write one group for the training crawlers and a separate group for the fetchers that act for a user in the moment, then keep a wildcard group for everything unnamed. `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended` belong in the first. `ChatGPT-User`, `Claude-User`, and `PerplexityBot` belong in the second, with your docs allowed. --- ## You will need A site whose robots.txt you control, and Node 22 or later to run the matcher. The rules follow [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), which standardized the protocol that had been a convention since 1994.[^1] Take the agent names from the operators rather than from a blog post. Each of them publishes a crawler page, and each page changes: see [the OpenAI list](https://platform.openai.com/docs/bots), [the Anthropic list](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), and [the Google list](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers). ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A noindex meta tag](https://developers.google.com/search/docs/crawling-indexing/block-indexing) | You want a page fetched but kept out of a search index | The page has to be fetched for the tag to be read, so collection still happens | The goal is to stop the fetch rather than the listing | | [Edge blocking on user agent](https://developers.cloudflare.com/bots/) | A crawler ignores the file and you need the refusal enforced | Rules that can catch a browser sharing a token, and a support load when they do | The crawler respects robots.txt and no enforcement is needed | | [No rules at all](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1) | A small site with nothing you would mind seeing in a model | No control over bulk collection, and no signal about what you intended | You have private paths, or a view on training use | | [robots.txt groups](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) | Any public site, as the statement of intent every other layer refers back to | An honor system, so a crawler that ignores it is unaffected | The traffic is hostile rather than merely unwanted | Two of these are requests and two are enforcement. robots.txt and a meta tag say what you want, and a crawler that ignores them faces nothing. Edge rules actually stop traffic, and they cost you false positives: an agent that sets a browser user agent gets through, and a developer using a scripted client sometimes does not. Run both, and treat the file as the statement of record. ## Split the named agents into two groups The split is between collection for training and retrieval for a person who is waiting. ```text title="robots.txt" User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / User-agent: ChatGPT-User User-agent: Claude-User User-agent: PerplexityBot Allow: /docs/ Allow: /openapi.json Disallow: /account/ Disallow: /admin/ User-agent: * Disallow: /account/ Disallow: /admin/ ``` `Google-Extended` is the one most often misread.[^2] It governs whether your content feeds Gemini and the models behind it, and it has no effect on Googlebot's search crawling. Adding it does not remove you from search results, and removing it does not put you back in. The wildcard group is the one an unnamed crawler lands in, and a new agent name appears every few months. Keep that group permissive enough to stay useful and strict about the paths that are private, because it is the rule that governs every crawler you have not heard of. Order inside a group does not matter, and order between groups does not either. That surprises people who write the file expecting the first match to win. Specificity decides the group, and path length decides the rule, so moving lines around changes nothing.[^3] ## Run the rules before you deploy them Rule matching has two parts people get wrong: which group applies to an agent, and which rule inside that group wins. ```js title="match.mjs" export function isAllowed(text, agent, path) { const group = groupFor(parse(text), agent) let winner = { allow: true, path: '' } for (const rule of group.rules) { if (!path.startsWith(rule.path)) continue const longer = rule.path.length > winner.path.length const tie = rule.path.length === winner.path.length && rule.allow if (longer || tie) winner = rule } return winner.allow } ``` One group applies, not all of them. The most specific matching user-agent line wins, and the wildcard group is ignored entirely for an agent named anywhere else in the file. A team that adds a name to the strict group and expects the wildcard `Disallow` rules to still apply has opened the private paths to that one crawler. ## Check it worked Run every agent against every path that matters and read the grid. ```bash node demo.mjs ``` ```text output agent /docs/quickstart /openapi.json /account/billing /blog/post GPTBot disallow disallow disallow disallow ClaudeBot disallow disallow disallow disallow ChatGPT-User allow allow disallow allow Claude-User allow allow disallow allow PerplexityBot allow allow disallow allow Googlebot allow allow disallow allow ``` The first two rows are the collection crawlers, refused everywhere. The next three are the fetchers acting for a person, which keep the documentation and the OpenAPI description while losing the account area. The last row is search, unaffected by any of it, which is the outcome the blanket rule would have destroyed. ```bash node --test match.test.mjs ``` ```text output 1..5 # tests 5 # suites 0 # pass 5 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 110.650071 ``` ## When it goes wrong An assistant reports it cannot read your docs. The agent matched a group you meant for training crawlers, because its name contains a string you matched loosely. Match the full token, and test the exact name from your access logs. Search traffic drops after a robots change. A crawler was refused a path that search depends on, or a `Disallow` prefix matched more paths than anyone intended. Compare before and after with the matcher, path by path. The rules have no effect at all. The file is served from the wrong host or with an HTML media type. It must sit at the root of the exact scheme, host, and port, and be served as plain text. ## When not to do this Do not use robots.txt to hide anything. The file is public, and a list of paths you would rather nobody fetched is a map for anyone curious. Put authentication in front of private routes, and leave the file for crawler policy. Do not refuse the user-triggered fetchers to make a point about training. A developer asking an assistant how to call your API is the case your documentation exists for, and refusing it sends them to a competitor whose docs load. Keep that lane open. Do not copy a rule set from another site without reading it. Names go out of date, one agent split into two, and a stale list refuses a crawler that no longer exists while missing the one that replaced it. Re-read the operators' own pages before each edit to the file. Do not treat the file as a legal position. It records an intention, and it binds nobody who chooses to ignore it. If the use of your content matters commercially, the terms of service and the edge rules are where that argument is actually made. ## Related how-tos - [Contract test an API you do not own](/howto/contract-test-an-api-you-do-not-own) - [Negotiate page size between an API and its clients](/howto/negotiate-page-size-with-clients) ## Last verified Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed. [^1]: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) is dated September 2022, and its first author is M. Koster, who wrote the [original](https://www.robotstxt.org/orig.html) as a consensus of the robots mailing list on June 30, 1994. Twenty-eight years is a long apprenticeship for a text file. The RFC adds definition language, error handling and caching to the 1994 method, and sets the floor on any parsing limit a crawler imposes at 500 KiB. [^2]: [Google's crawler page](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended) states that `Google-Extended` has no HTTP user agent string of its own. Crawling is done under the existing Google user agents, and the token is, in the page's words, used in a control capacity. Nothing on the wire ever announces itself by that name, so an access log never shows it. The example group the page gives allows `/archive/1Q84` and disallows the rest of `/archive/`, which is the most specific match rule in two lines. [^3]: Length in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.2) is counted in octets, the word the RFC uses, rather than in characters or lines. The most specific match must be used, and where an `allow` and a `disallow` are equivalent the `allow` should win, which is why the matcher on this page breaks ties that way. The same section adds that duplicate rules in a group may be deduplicated, and that a group with no rules allows everything.