How to measure drift between two AI-written clients#

Two engineers ask an agent for a client for the same API on the same afternoon. One client retries on a timeout and paginates; the other does neither and skips an endpoint entirely. Both pass review, because reviewing a client means reading its own diff, which says nothing about what the other one did.

What you get

You will end up with a structured comparison of two client surfaces, reporting coverage gaps, dropped parameters, differing error handling, and behavior neither prompt specified. This is for you if agents write integration code in your codebase.

Short answer

Describe each client as data: which operations it covers, which parameters it passes, which statuses it handles, and which behaviors such as retry and pagination it implements. Diff the two descriptions. Differences are drift, and anything neither client covers is a gap in the brief rather than a failure of either writer.

You will need

Node 22 or later, and two clients written from the same description. The surface description can be extracted rather than written. A walk over the exported functions and their call sites gives you the operations and the parameters. The TypeScript compiler API is one way to do it,¹ and ts-morph is a friendlier wrapper over the same thing.

Approaches compared

The comparison and the test suite answer different questions. A diff of the surfaces says one client never implemented an endpoint; a contract suite says whether what was implemented behaves. Neither substitutes for the other, and the diff is far cheaper to build first.

The finding that repeats across teams is the last category. Retry, pagination, and timeout handling are decided by whichever agent wrote the file, because the description says nothing about them. Those are not model failures. They are decisions nobody made, and the diff is the cheapest place to notice that.

Describe each client as data

You can extract all four fields of each operation from the code.

{
  "writer": "agent-b",
  "operations": [
    { "id": "listMeters", "params": ["cursor"], "handles": [429], "retries": true, "paginates": false },
    { "id": "loadMeter", "params": ["id"], "handles": [404], "retries": false, "paginates": false },
    { "id": "createMeter", "params": [], "handles": [400, 409], "retries": true, "paginates": false }
  ]
}

Extract rather than annotate. A description a person maintains by hand describes the client they remember, and the whole point is to find out what the code does. Extraction also scales: the fifth client costs nothing to add once the walk exists.

Include the writer’s name. Once you have several of these, the question stops being whether two clients agree and becomes which prompts produce clients that handle rate limits.

Separate drift from a gap in the brief

Two functions, because the two findings need different responses.

export function unhandledErrors(spec, ...clients) {
  const gaps = []
  for (const op of spec.operations) {
    for (const status of op.errors) {
      const covered = clients.some((c) => c.operations.find((o) => o.id === op.id)?.handles.includes(status))
      if (!covered) gaps.push(`${op.id} ${status}`)
    }
  }
  return gaps
}

A difference between the clients is a review question. Something neither client does is a prompt question that no amount of comparing the two will reveal, because they agree.

Feed the gaps back into the brief rather than into a fix. Patching one client leaves the next one with the same hole, and the next one after that.

Check it worked

Compare two clients written from one description.

node demo.mjs

7 findings across 4 operations
  parameters  listMeters   agent-b omits limit
  errors      listMeters   handled differently: 400
  behavior    listMeters   paginates: agent-a=true, agent-b=false
  behavior    loadMeter    retries: agent-a=true, agent-b=false
  errors      createMeter  handled differently: 409
  behavior    createMeter  retries: agent-a=false, agent-b=true
  coverage    retireMeter  missing from agent-b

not covered by either client
  listMeters 503

Three of the seven findings are behavior, and each one is a decision the description never made. One client paginates and one does not, and the one that retries a read does not retry a create, while the other does the opposite. Each is defensible on its own, but only one can be your house style.

The second block matters more than the first. The two clients agree on one documented failure by both leaving it unhandled, so no comparison between them would ever have found it. That is the case a review of the diff cannot reach.

Say “neither” and mean it. The predicate is some, not every.² A status one client handles and the other misses is drift, and the first block already reports it. Counting it here as well turns a short list of prompt fixes into a long one, and sends you to rewrite a description that was never the problem. Getting this backwards is easy, because the report still looks plausible.

That split is the reason to build the second function. Drift is a review conversation and a gap is a prompt change, and mixing them produces a report where the fixable items are outnumbered by the arguable ones.

node --test drift.test.mjs

1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 113.99799

When it goes wrong

Everything reads as drift. The extraction is picking up formatting rather than behavior. Model the surface at the level you would review, not at the level of the syntax tree. Whitespace, import order and the choice between a class and a closure are not drift, and a report full of them gets ignored within a week.

The diff is empty and the clients behave differently. The model misses what differs, usually retries or timeouts hidden in a wrapper. Add a field for each behavior you care about and extract it explicitly. Retry, timeout, pagination and error mapping are the four worth having from the start.

Findings pile up without changing anything. There is no owner for the house conventions the clients keep disagreeing about. Write those conventions down, put them in the prompt, and re-measure. A convention that lives only in review comments gets re-litigated on every client.

The two clients agree on a behavior that is wrong in both, because two runs of the same model share the same blind spots. Agreement is not correctness. Keep a contract test suite as the other half, and write its cases from the description rather than from either client.

When not to do this

Do not compare clients written from different descriptions. The findings will be real differences in the APIs, and the report will bury the ones that matter. Pin the description version into each surface file so a mismatch is visible.

Do not use the comparison to pick a winner. The useful output is a list of decisions nobody made, and turning it into a score sends people to tune prompts instead of writing down conventions.

Do not skip the contract tests because the diff is clean. A surface comparison cannot see a wrong header, a mishandled encoding, or a retry that repeats a write.

Related how-tos

• Contract test an API you do not own

• Decide between generating and hand-writing a client

Last verified

Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.