# How to cache tool results across the steps of one agent run > Cache an agent's read tools by tenant, tool and arguments, clear the cache on every write, and record requests saved beside whether the answer stayed right. Source: https://voxgig.com/howto/cache-tool-results-across-agent-steps - Audience: agent-builder - Level: intermediate - Languages: python, typescript - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Put a memo in front of the agent's read tools, keyed by the caller's tenant, the tool name and the arguments with their keys sorted, and give entries a lifetime of seconds. Clear the tenant's entries whenever a write tool runs, and drop the memo when the run ends. Record hits beside task success on every run, so a cache that changes answers shows up as a falling success rate rather than a rising hit rate. --- ## You will need Node 22 or later, Python 3.7 or later for the Python twin, and an agent whose tools call an API. Count tokens and cost per task before you start, so the saving lands in a number you already track. Verified 2026-10-15 against Node 22.22.0, Python 3.13.16, undici 7.30.1, @voxgig/create-sdkgen 0.30.7 and @voxgig/sdkgen 4.34.7. A scripted model stub drives every run, so no provider is called, and the two-tenant ticket API is a local server each command starts. Voxgig maintains sdkgen. This page compares its cache feature with a per-run memo, HTTP caching through undici, and Redis. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A per-run memo in the agent loop](https://docs.python.org/3/library/functools.html#functools.lru_cache) | Repeated reads inside one run, with writes the loop can see | Code you own: the key, the lifetime, and clearing it on writes | Runs repeat each other's reads, so a shared store saves more | | [HTTP caching with `ETag` and `Cache-Control`](https://github.com/nodejs/undici/blob/v7.x/docs/docs/api/Dispatcher.md) | The API sends caching headers and should decide what stays fresh | It caches only what the server allows, and tenants stay apart only while it sends `Vary` | The API sends no caching headers at all | | [Redis](https://redis.io/docs/latest/commands/set/) | Many runs and workers repeat the same reads across processes | A server to run, and every key must carry the tenant because every run shares it | One run at a time, where a memo costs nothing | | [The sdkgen cache feature](https://voxgig.com/sdk/features) | A client generated by sdkgen, over reads that rarely change | Plain time-to-live, 5000 ms by default, keyed by method and URL, blind to headers and to your own writes | Reads that your own writes make stale within the lifetime | The memo knows about the run, so it can drop entries when a write happens, but it cannot see what the server considers fresh. HTTP caching is the reverse: it obeys the server's headers and cannot see which reads matter to the task. Redis extends the memo across runs and multiplies the cost of a key that forgets the tenant. The sdkgen feature is one option in the client constructor, with the limits measured below. ## Key the memo by tenant, tool and arguments The key starts with the tenant, taken from the client that sends the request rather than from anything the model wrote. The same tool and arguments return different records for different callers, so a key whose tenant came from a second source can file one tenant's record under a different tenant. ```js title="cache.mjs" async through(tenant, tool, args, call) { if (!tenant) throw new Error('a cached tool call needs the caller identity') const key = this.keyOf(tenant, tool, args) const entry = this.store.get(key) if (entry && entry.expires > this.now()) { this.hits++ return structuredClone(entry.value) } // Parallel calls for the same key share one request instead of racing to make two. if (this.inflight.has(key)) { this.hits++ return structuredClone(await this.inflight.get(key)) } this.misses++ const started = this.generation.get(tenant) ?? 0 const pending = call() this.inflight.set(key, pending) try { const value = await pending // A write that landed while this read was in flight makes its answer suspect: return it, // but do not keep it. if ((this.generation.get(tenant) ?? 0) === started) { this.store.set(key, { value: structuredClone(value), expires: this.now() + this.ttlMs, tenant }) } return value } finally { this.inflight.delete(key) } } ``` Each of three details has a test that fails when the detail is removed. `structuredClone` stops a caller that edits a returned record from editing the cache. The in-flight map makes parallel tool calls for one customer cost one request. The generation counter keeps a read that was in flight when a write landed from being stored. Instances that share a store share its counter, so the write can come through any of them. The tools decide what is cached. Reads go through the memo; a write clears the tenant's entries in a `finally`, because a write that failed may still have changed something. ```js title="tools.mjs" export function makeTools(sdk, { cache, invalidateOnWrite = true } = {}) { // The key's tenant is the one this client sends. Taken from anywhere else, it can name acme // while the request fetches globex's record, and acme's next read is served globex's. const tenant = sdk.options().headers?.['x-tenant'] const read = (tool, fn) => (args) => (cache ? cache.through(tenant, tool, args, () => fn(args)) : fn(args)) const write = (fn) => async (args) => { try { return await fn(args) } finally { if (cache && invalidateOnWrite) cache.invalidate(tenant) } } ``` ## Measure the saving beside the answer The scripted task closes ticket `t1` if its customer is on the free plan, then confirms the status. It lists three tickets, looks up their customer three times, reads `t1`, writes it, and reads it again. Each row is the same seven tool calls with a different cache below them. ```bash node measure.mjs ``` ```text output approach tool calls requests saved answer right no cache 7 7 0% t1 is closed yes per-run memo 7 5 29% t1 is closed yes memo, writes ignored 7 4 43% t1 is open NO sdkgen cache feature 7 4 43% t1 is open NO HTTP cache (undici) 7 5 29% t1 is closed yes ``` The two highest savings come with the wrong answer. Both served the confirming read from an entry made before the write, so the agent reported `t1 is open` about a ticket it had just closed. A dashboard of hit rates alone would have ranked those two first. The memo and the HTTP cache saved the same two customer reads by different routes. The memo cleared itself on the write. The HTTP cache sent a conditional request for the ticket, which the server marks `no-cache`. Its handler in undici 7.30.1 also drops a stored URL when an unsafe method against it succeeds, as [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111.html#section-4.4) requires. The Python twin has the same shape, without the in-flight handling a synchronous loop does not need. Python's `functools.lru_cache` counts hits and misses through `cache_info()`, but it has no lifetime of its own and `cache_clear()` empties it for every tenant at once, so the twin keeps a dictionary. ```python title="memo.py" def key(self, tenant, tool, args): parts = [tenant, tool] if self.with_tenant else [tool] return json.dumps(parts + [args], sort_keys=True) def through(self, tenant, tool, args, call): if not tenant: raise ValueError("a cached tool call needs the caller identity") key = self.key(tenant, tool, args) entry = self.store.get(key) if entry and entry["expires"] > self.clock(): self.hits += 1 return copy.deepcopy(entry["value"]) self.misses += 1 value = call() self.store[key] = {"value": copy.deepcopy(value), "expires": self.clock() + self.ttl, "tenant": tenant} return value ``` ```bash python3 memo.py ``` ```text output answer: t1 is closed; api calls 4; hits 2, misses 3, hit rate 40% no tenant in the key: acme sees Acme Rail, globex sees Acme Rail tenant in the key: acme sees Acme Rail, globex sees Globex ``` ## Check it worked Fourteen tests pin the measurements and the edges: expiry on an injected clock, argument order, parallel calls, and a read racing a write. The write may come through the same instance or another on the same store. They also cover a failed write that still clears the memo, the tenant taken from the client, and the sdkgen feature's 5000 ms default. ```bash node --test cache.test.mjs ``` ```text output 1..14 # tests 14 # suites 0 # pass 14 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 421.183628 ``` ## When it goes wrong Globex's agent reads Acme's customer. The cache key has no tenant, takes it from somewhere other than the request, or the tenant travels in a header the cache never reads. Put the tenant the client sends first in every key. ```bash node pitfall.mjs ``` ```text output two reads of t1, served with Cache-Control: no-cache, an ETag and Last-Modified sdkgen cache feature requests 1 HTTP cache (undici) requests 2 304 Not Modified answers sent: 1 customer c1 for acme, then for globex, through one cache memo keyed by tool and arguments acme: Acme Rail, globex: Acme Rail memo keyed by tenant, tool and arguments acme: Acme Rail, globex: Globex sdkgen cache, one client, tenant per call acme: Acme Rail, globex: Acme Rail sdkgen cache, one client per tenant acme: Acme Rail, globex: Globex HTTP cache shared, server sends Vary acme: Acme Rail, globex: Globex ``` The sdkgen cache keys on method and URL, so one client that sends the tenant per call serves the first tenant's record to the second. Each client holds its own store, so a client per tenant is safe. Stripe's Connect API works this way, with the account in a header on each request.[^1] The server says `no-cache` and the cache serves the record anyway. The sdkgen feature reads no caching headers, so its second read never reached the server. undici sent a conditional request and got `304 Not Modified`. It stored that response only once the server added `Last-Modified`.[^2] The success rate drops after you add the cache. A write made the cached read stale. Clear the tenant's entries on every write tool, or shorten the lifetime below the gap between a write and the read that checks it. ## When not to do this Do not turn on sdkgen's cache feature for an agent that reads what it has just written. In 4.34.7 its lifetime, 5000 ms unless you set `ttl`, is the only thing that expires an entry. A write through the same client clears nothing, which the measurement on this page shows as a wrong answer. It suits reference data that changes less often than the lifetime. Do not share a cache across tenants unless every key carries the tenant. In Redis that means the key string itself, set with an `EX` lifetime, because a shared store serves every caller. Do not cache the result of a write, an error, or a read made while a write was in flight. None of them describes the state the next step will see. Do not report a hit rate without the task success beside it. ## Related how-tos - [Build cache keys that include the caller and Vary headers](/howto/cache-keys-that-include-the-caller) - [Share an API response cache across many workers with Redis](/howto/share-a-response-cache-across-workers-with-redis) - [Cut agent cost with provider prompt caching](/howto/cut-agent-cost-with-prompt-caching) - [Add a dry-run mode to an agent's write tools](/howto/add-dry-run-to-agent-write-tools) ## Last verified Verified 2026-10-15 against Node 22.22.0, Python 3.13.16, undici 7.30.1, @voxgig/create-sdkgen 0.30.7 and @voxgig/sdkgen 4.34.7. Every output block is what the command preceding it printed. The generated SDK's compiled JavaScript is committed under `generated/`, and `README.md` beside it names the commands that produced it. Redis was not run; its row rests on its documentation. [^1]: The [Connect authentication guide](https://docs.stripe.com/connect/authentication) tells a platform to include the connected account's ID in the `Stripe-Account` header on each request, an ID that begins with `acct_`. Every connected account on a platform is served from the same URLs. The header is the whole of the difference between one business's records and the next. [^2]: With no `max-age` and no `Expires`, undici falls back to the heuristic in [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111.html#name-calculating-heuristic-fresh). A response with a `Last-Modified` date may be considered fresh for a fraction of the time since that date, and the RFC suggests that a typical setting of this fraction might be 10%. undici's [cache interceptor documentation](https://github.com/nodejs/undici/blob/v7.x/docs/docs/api/Dispatcher.md) says that a response with neither an explicit expiration nor a heuristic one is not cached unless `cacheByDefault` is set. The same RFC section records that an earlier specification, RFC 2616, forbade heuristic freshness for any URI with a query string, and observes that in practice this was not widely implemented. A rule that nobody followed has been retired with a note saying so, which is more ceremony than most rules get.