# How to measure drift between two AI-written clients > Compare two clients written from the same description and separate what one of them missed from what neither was told. Source: https://voxgig.com/howto/measure-drift-between-two-ai-written-clients - Audience: agent-builder - Level: intermediate - Languages: typescript, javascript - Verified: 2026-09-14 - Published: 2026-09-14 - Updated: 2026-09-24 ## Short answer Describe each client as data: which operations it covers, which parameters it passes, which statuses it handles, and which behaviors such as retry and pagination it implements. Diff the two descriptions. Differences are drift, and anything neither client covers is a gap in the brief rather than a failure of either writer. --- ## You will need Node 22 or later, and two clients written from the same description. The surface description can be extracted rather than written. A walk over the exported functions and their call sites gives you the operations and the parameters. [The TypeScript compiler API](https://github.com/microsoft/TypeScript/wiki/Using-the-Compiler-API) is one way to do it,[^1] and [ts-morph](https://ts-morph.com/) is a friendlier wrapper over the same thing. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A structured diff of both surfaces](https://nodejs.org/api/test.html) | You want the comparison repeatable and the findings categorized | An extraction step, and a model of what counts as a surface | The clients differ in ways your model does not capture | | [Reading both diffs by hand](https://git-scm.com/docs/git-diff) | Two small clients, once | Attention, which runs out around the third file and does not scale | This happens more than once a quarter | | [Regenerating until they agree](https://www.anthropic.com/engineering/claude-code-best-practices) | You would rather fix the prompt than compare the output | Nondeterminism, so agreement on one run says little about the next | You need to know what differed and why | | [Running both against a contract test suite](https://docs.pact.io/) | The behavior matters more than the shape, which is usually true | A test suite covering the cases, which is the work the clients were meant to save | You care about coverage of the description itself | The comparison and the test suite answer different questions. A diff of the surfaces says one client never implemented an endpoint; a contract suite says whether what was implemented behaves. Neither substitutes for the other, and the diff is far cheaper to build first. The finding that repeats across teams is the last category. Retry, pagination, and timeout handling are decided by whichever agent wrote the file, because the description says nothing about them. Those are not model failures. They are decisions nobody made, and the diff is the cheapest place to notice that. ## Describe each client as data You can extract all four fields of each operation from the code. ```json title="client-b.json" { "writer": "agent-b", "operations": [ { "id": "listMeters", "params": ["cursor"], "handles": [429], "retries": true, "paginates": false }, { "id": "loadMeter", "params": ["id"], "handles": [404], "retries": false, "paginates": false }, { "id": "createMeter", "params": [], "handles": [400, 409], "retries": true, "paginates": false } ] } ``` Extract rather than annotate. A description a person maintains by hand describes the client they remember, and the whole point is to find out what the code does. Extraction also scales: the fifth client costs nothing to add once the walk exists. Include the writer's name. Once you have several of these, the question stops being whether two clients agree and becomes which prompts produce clients that handle rate limits. ## Separate drift from a gap in the brief Two functions, because the two findings need different responses. ```js title="drift.mjs" export function unhandledErrors(spec, ...clients) { const gaps = [] for (const op of spec.operations) { for (const status of op.errors) { const covered = clients.some((c) => c.operations.find((o) => o.id === op.id)?.handles.includes(status)) if (!covered) gaps.push(`${op.id} ${status}`) } } return gaps } ``` A difference between the clients is a review question. Something neither client does is a prompt question that no amount of comparing the two will reveal, because they agree. Feed the gaps back into the brief rather than into a fix. Patching one client leaves the next one with the same hole, and the next one after that. ## Check it worked Compare two clients written from one description. ```bash node demo.mjs ``` ```text output 7 findings across 4 operations parameters listMeters agent-b omits limit errors listMeters handled differently: 400 behavior listMeters paginates: agent-a=true, agent-b=false behavior loadMeter retries: agent-a=true, agent-b=false errors createMeter handled differently: 409 behavior createMeter retries: agent-a=false, agent-b=true coverage retireMeter missing from agent-b not covered by either client listMeters 503 ``` Three of the seven findings are behavior, and each one is a decision the description never made. One client paginates and one does not, and the one that retries a read does not retry a create, while the other does the opposite. Each is defensible on its own, but only one can be your house style. The second block matters more than the first. The two clients agree on one documented failure by both leaving it unhandled, so no comparison between them would ever have found it. That is the case a review of the diff cannot reach. Say "neither" and mean it. The predicate is `some`, not `every`.[^2] A status one client handles and the other misses is drift, and the first block already reports it. Counting it here as well turns a short list of prompt fixes into a long one, and sends you to rewrite a description that was never the problem. Getting this backwards is easy, because the report still looks plausible. That split is the reason to build the second function. Drift is a review conversation and a gap is a prompt change, and mixing them produces a report where the fixable items are outnumbered by the arguable ones. ```bash node --test drift.test.mjs ``` ```text output 1..6 # tests 6 # suites 0 # pass 6 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 113.99799 ``` ## When it goes wrong Everything reads as drift. The extraction is picking up formatting rather than behavior. Model the surface at the level you would review, not at the level of the syntax tree. Whitespace, import order and the choice between a class and a closure are not drift, and a report full of them gets ignored within a week. The diff is empty and the clients behave differently. The model misses what differs, usually retries or timeouts hidden in a wrapper. Add a field for each behavior you care about and extract it explicitly. Retry, timeout, pagination and error mapping are the four worth having from the start. Findings pile up without changing anything. There is no owner for the house conventions the clients keep disagreeing about. Write those conventions down, put them in the prompt, and re-measure. A convention that lives only in review comments gets re-litigated on every client. The two clients agree on a behavior that is wrong in both, because two runs of the same model share the same blind spots. Agreement is not correctness. Keep a contract test suite as the other half, and write its cases from the description rather than from either client. ## When not to do this Do not compare clients written from different descriptions. The findings will be real differences in the APIs, and the report will bury the ones that matter. Pin the description version into each surface file so a mismatch is visible. Do not use the comparison to pick a winner. The useful output is a list of decisions nobody made, and turning it into a score sends people to tune prompts instead of writing down conventions. Do not skip the contract tests because the diff is clean. A surface comparison cannot see a wrong header, a mishandled encoding, or a retry that repeats a write. ## Related how-tos - [Contract test an API you do not own](/howto/contract-test-an-api-you-do-not-own) - [Decide between generating and hand-writing a client](/howto/generate-or-hand-write-a-small-api-client) ## Last verified Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed. [^1]: The wiki page carries a caution at the top. It describes TypeScript 6.0 and earlier, it says, and [TypeScript 7.1 will have a completely different API](https://raw.githubusercontent.com/wiki/microsoft/TypeScript/Using-the-Compiler-API.md). The setup instructions accordingly ask for a version of at least 1.6 and below 7. [ts-morph](https://ts-morph.com/) describes itself as a wrapper that exists because setup, navigation, and manipulation of the TypeScript syntax tree can be a challenge. A wrapper over an API that is about to be replaced is a bet, and the page that documents the API says which way. [^2]: The two predicates disagree about nothing. The MDN page for [`every`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/every) says that for an empty array it returns true, since it is vacuously true that all elements of the empty set satisfy any condition. The page for [`some`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/some) says it returns false for any condition. Run the gap finder with no clients and `some` reports every documented error as uncovered, which is correct. `every` would report a clean sheet, which is the other kind of correct.