How to write an AGENTS.md for the root of a documentation site#

An agent is sent to your product with nothing but your docs URL. It finds a landing page built for people: a search box, cards, a sidebar, and a quickstart that opens with account creation. Twenty requests later it is calling a path it found in a blog post, because nothing it read said which URLs are stable, how to authenticate, or what not to try.

What you get

You will end up with an AGENTS.md under fifty lines, served as plain text, and a checker that resolves every URL in it and runs every command. This is for you if agents arrive at your docs with one URL and a task.

Short answer

Publish AGENTS.md at the root of your docs site as plain text, under 100 lines. Put the stable URLs an agent needs first, then the install command, one worked call with its response, auth and rate limits, and what not to attempt. Link llms.txt instead of repeating its index. Then fetch the file as an agent would, resolve every URL it names, run every command in a clean shell, and fail CI when any of them breaks.

You will need

Node 22 or later for the checker, and a documentation site whose root you can serve a file from. Verified 2026-10-21 against Node 22.22.0. The sample serves a stand-in site on 127.0.0.1 with every file the AGENTS.md names, so the page runs offline. The convention this borrows is AGENTS.md, the Markdown file with no required fields that coding agents read in a repository. Serving one at a site root extends it to the product, as voxgig.com does.¹

Approaches compared

llms.txt and AGENTS.md overlap less than their names suggest. The llms.txt proposal is an index: an H1, a summary, and lists of links with a note on each, so an agent can choose what to read.² AGENTS.md is instructions: what to run and what to avoid. Keep both and cross-link them rather than duplicating, because the index goes stale page by page and the instructions go stale route by route, and each needs a different check.

Put the stable URLs first, then the install, then one call

An agent reads top down and may stop early, so the first screen carries what it needs to act. That is the URLs that will not move, the one command that installs the client, and one call with the response it should expect. Auth and limits follow, then the list of things not to attempt, and everything else is a link. The fixture is 48 lines, and opens with the URLs:

## Stable URLs

- OpenAPI document: https://docs.meterco.example/openapi.json
- Page index for agents: https://docs.meterco.example/llms.txt
- MCP endpoint, Streamable HTTP, POST only: https://docs.meterco.example/mcp
- Health: https://docs.meterco.example/api/health
- API base: https://api.meterco.example/v2

Next come the install command, npm install @meterco/sdk@2.4.0, and one worked call: a curl -sS --fail with the key in a bearer token, followed by the JSON line it returns. Then the two sections that keep an agent out of trouble:

## Auth and limits

- Keys start with `mk_live_` or `mk_test_`. Create one at https://docs.meterco.example/keys.
- 60 requests per minute per key. A 429 carries `Retry-After` in seconds; wait that long.
- Every error is an RFC 9457 problem document, never an HTML page.

## Do not

- Do not call anything under `/v2/admin/`; those routes answer 403 to every key but the owner's.
- Do not poll `/readings/latest` more than once a minute; readings arrive every fifteen minutes.
- Do not scrape the HTML docs; every page has a markdown twin at its path plus `.md`.

Three choices in that file carry weight. Every URL is absolute, because the agent reads the file out of context and pastes from it. The worked call shows the response, so the agent can tell a success from a problem document before it writes a parser, and the file says the errors follow RFC 9457. The curl carries --fail, so a missing key turns into a non-zero exit for the next step to catch. The llms.txt beside it points back rather than repeating:

# Meterco

> Meterco meters electricity use per site and exposes the readings over a REST API, an MCP
> endpoint, and an SDK for Node.

Instructions for agents, with one worked call, are at https://docs.meterco.example/AGENTS.md.

Check the file the way an agent reads it

The checker fetches the URL and asks four questions. Is the response plain text, meaning a Content-Type of text/markdown or text/plain rather than HTML. Does every URL in the file resolve, where a 405 from a POST-only MCP endpoint and a 401 from a route that wants the key both count as present. Does every command in a bash fence exit zero in a shell holding only PATH, HOME, and the variables you pass, which is what env -i gives you at a prompt.³ And do the five parts sit inside the first 100 lines.

  for (const found of urlsIn(text)) {
    const target = mapOrigins(found, as, base)
    let status = 0
    let ctype = ''
    try {
      const r = await fetchImpl(target, { method: 'GET', redirect: 'follow' })
      status = r.status
      ctype = (r.headers.get('content-type') ?? '').split(';')[0]
      await r.arrayBuffer()
    } catch (e) {
      status = `unreachable (${e.cause?.code ?? e.message})`
    }
    // 405 is a route that exists and refuses GET, which is what a POST-only MCP endpoint says,
    // and 401 is a route that exists and wants the key the file tells the agent to send.
    const ok = status === 200 || status === 401 || status === 405

mapOrigins makes the check usable before a deploy: --as https://docs.meterco.example rewrites that origin onto the server under check, so staging is checked against production’s URLs. The demo starts the stand-in site, points the install at a one-package npm registry the same server hosts through npm_config_registry, and passes the key:

node demo.mjs

  200 text/markdown; charset=utf-8  48 lines  1603 bytes
links
  ok   200  /AGENTS.md                         text/markdown
  ok   200  /openapi.json                      application/json
  ok   200  /llms.txt                          text/plain
  ok   405  /mcp                               application/json
  ok   200  /api/health                        application/json
  ok   200  /v2                                application/json
  ok   401  /v2/meters/mtr_1/readings/latest   application/problem+json
  ok   200  /keys                              text/markdown
  ok   200  /reference                         text/markdown
  ok   200  /changelog                         text/markdown
commands, in a shell with only PATH, HOME, and the variables you passed
  ok   exit 0  npm install @meterco/sdk@2.4.0
  ok   exit 0  curl -sS --fail -H "Authorization: Bearer $METERCO_KEY" https://api.meterco.example/v2/meters/mtr_1/readings/latest
first 100 lines carry 5 of 5 parts: urls, install, worked call, auth and limits, do not
result: ok

Read the two non-200 rows as the point of the walk. The MCP endpoint answers 405 to a GET because it takes POST. The worked call’s URL answers 401 to a bare GET because it wants the key; a 404 in either place would be the file lying. The install ran against a registry on 127.0.0.1, so its exit code says nothing about the public registry; drop --registry in CI, where the network is there.

Give an agent the URL and one task

The checker proves the file is true, and only an agent can say whether it is sufficient. Give one nothing but the URL and a task the file should make possible:

claude -p "Read https://docs.meterco.example/AGENTS.md, then fetch the latest reading for meter mtr_1 and show me the exact request you sent."

Look at the request, not the prose. The path has to be the one in the worked call, with the key as a bearer token, and /v2/admin/ left alone. A run that reaches for the quickstart first says the file is missing something. This run needs a model and an account, so this page did not run it.

Check it worked

Nine tests cover the four questions and the shapes that fail them.

node --test agents-md.test.mjs

1..9
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0

Two matter most. One serves the same file as text/html and asserts that the media type is the first finding, whatever the body says. The other runs the worked call with the key and without, and asserts exit 0 then exit 22, which is what curl --fail returns on the 401 the missing key causes. The test exports a key in its own environment first, so a checker that leaked the session into the shell would see exit 0 twice.

When it goes wrong

The agent never reaches the limits. The quickstart was pasted into the file, so the parts an agent needs most sit below a hundred lines of account creation and Node installation. The pitfall script measures the fixture both ways:

node pitfall.mjs

site-root AGENTS.md: 48 lines, 1603 bytes
  first URL at line 4, first command at line 18
  urls             line   7
  install          line  15
  worked call      line  21
  auth and limits  line  33
  do not           line  39
quickstart pasted in: 196 lines, 6024 bytes
  first URL at line 152, first command at line 40
  install          line  34
  auth and limits  line  75
  worked call      line  96
  urls             line 155  past the first 100 lines
  do not           line 187  past the first 100 lines
checker on the pasted version: FAIL
  finding: urls starts at line 155, past the first 100 lines
  finding: do not starts at line 187, past the first 100 lines

The pasted version still has an install heading at line 34, because the quickstart installs Node there. Its worked call at line 96 is a JavaScript file rather than a request, and the first stable URL arrives at line 152. Keep the quickstart as a link, and keep the budget check in CI so the next paste fails the build rather than the agent.

The checker’s first finding is content-type is text/html. The docs framework answers every unknown path with its single-page shell, so /AGENTS.md is a page of navigation with no Markdown in it. Serve the file as a static asset with an explicit type, as the stand-in does with text/markdown; charset=utf-8, or add a route for it that sets the header.

Every command passes and the key was never set. The worked call used curl -s without --fail, so curl printed the problem document and exited 0. Add --fail to every curl in the file; the test suite asserts the exit code so the omission cannot come back.

When not to do this

Do not write a site-root AGENTS.md for a product with nothing to call. A pure documentation site has no worked call to show, and llms.txt plus markdown twins are its agent surface.

Do not copy the repository’s AGENTS.md to the site root. That file tells an agent how to build and test the code, which misleads an agent that wants to call the product and names commands only a checkout has. Two files, two audiences.

Do not list a URL you will not keep stable. An agent given a URL here keeps using it, so a route that moves is called at its former address from code nobody revisits. Leave the route out, or publish it under a versioned path, as a stable OpenAPI URL does.

Do not ship the file without the checker in CI. Every line in it is a claim about a URL, a package, or a limit, and each of them can change without anyone editing the file.

Do not point the checker at a file you do not publish. It runs every command in the file’s bash fences with the permissions of whoever runs it, and the clean environment withholds your variables from those commands without confining what they do.

Related how-tos

• Publish an OpenAPI document at a stable URL for agents

• Add a health endpoint agents can poll

• Make a Docusaurus site agent-ready

• Set robots.txt rules for each AI crawler

Last verified

Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The site, the API, and the npm registry the install command reaches are stand-ins on 127.0.0.1 started by each script, so the install and the worked call run with no network. The agent run in the third step was not run, because it needs a model and an account. The voxgig.com file was fetched once by hand for its headers and length, outside the gate.