How to cache tool results across the steps of one agent run#

Your agent looks up the same customer three times in one run, once for each ticket it reads, and each lookup is a billed API call and a slower step. Cache those results naively and the run’s last read, made after its own write, returns the record as it was before the write. The agent then reports a status that is no longer true.

What you get

You will end up with a per-run cache for tool results, keyed by tenant, tool and arguments, and a measurement that puts requests saved beside whether the answer stayed right. This is for you if your agent makes the same read more than once in a run.

Short answer

Put a memo in front of the agent’s read tools, keyed by the caller’s tenant, the tool name and the arguments with their keys sorted, and give entries a lifetime of seconds. Clear the tenant’s entries whenever a write tool runs, and drop the memo when the run ends. Record hits beside task success on every run, so a cache that changes answers shows up as a falling success rate rather than a rising hit rate.

You will need

Node 22 or later, Python 3.7 or later for the Python twin, and an agent whose tools call an API. Count tokens and cost per task before you start, so the saving lands in a number you already track. Verified 2026-10-15 against Node 22.22.0, Python 3.13.16, undici 7.30.1, @voxgig/create-sdkgen 0.30.7 and @voxgig/sdkgen 4.34.7. A scripted model stub drives every run, so no provider is called, and the two-tenant ticket API is a local server each command starts.

Voxgig maintains sdkgen. This page compares its cache feature with a per-run memo, HTTP caching through undici, and Redis.

Approaches compared

The memo knows about the run, so it can drop entries when a write happens, but it cannot see what the server considers fresh. HTTP caching is the reverse: it obeys the server’s headers and cannot see which reads matter to the task. Redis extends the memo across runs and multiplies the cost of a key that forgets the tenant. The sdkgen feature is one option in the client constructor, with the limits measured below.

Key the memo by tenant, tool and arguments

The key starts with the tenant, taken from the client that sends the request rather than from anything the model wrote. The same tool and arguments return different records for different callers, so a key whose tenant came from a second source can file one tenant’s record under a different tenant.

  async through(tenant, tool, args, call) {
    if (!tenant) throw new Error('a cached tool call needs the caller identity')
    const key = this.keyOf(tenant, tool, args)
    const entry = this.store.get(key)
    if (entry && entry.expires > this.now()) {
      this.hits++
      return structuredClone(entry.value)
    }
    // Parallel calls for the same key share one request instead of racing to make two.
    if (this.inflight.has(key)) {
      this.hits++
      return structuredClone(await this.inflight.get(key))
    }
    this.misses++
    const started = this.generation.get(tenant) ?? 0
    const pending = call()
    this.inflight.set(key, pending)
    try {
      const value = await pending
      // A write that landed while this read was in flight makes its answer suspect: return it,
      // but do not keep it.
      if ((this.generation.get(tenant) ?? 0) === started) {
        this.store.set(key, { value: structuredClone(value), expires: this.now() + this.ttlMs, tenant })
      }
      return value
    } finally {
      this.inflight.delete(key)
    }
  }

Each of three details has a test that fails when the detail is removed. structuredClone stops a caller that edits a returned record from editing the cache. The in-flight map makes parallel tool calls for one customer cost one request. The generation counter keeps a read that was in flight when a write landed from being stored. Instances that share a store share its counter, so the write can come through any of them.

The tools decide what is cached. Reads go through the memo; a write clears the tenant’s entries in a finally, because a write that failed may still have changed something.

export function makeTools(sdk, { cache, invalidateOnWrite = true } = {}) {
  // The key's tenant is the one this client sends. Taken from anywhere else, it can name acme
  // while the request fetches globex's record, and acme's next read is served globex's.
  const tenant = sdk.options().headers?.['x-tenant']
  const read = (tool, fn) => (args) => (cache ? cache.through(tenant, tool, args, () => fn(args)) : fn(args))
  const write = (fn) => async (args) => {
    try {
      return await fn(args)
    } finally {
      if (cache && invalidateOnWrite) cache.invalidate(tenant)
    }
  }

Measure the saving beside the answer

The scripted task closes ticket t1 if its customer is on the free plan, then confirms the status. It lists three tickets, looks up their customer three times, reads t1, writes it, and reads it again. Each row is the same seven tool calls with a different cache below them.

node measure.mjs

approach              tool calls  requests  saved  answer        right
no cache              7           7         0%     t1 is closed  yes
per-run memo          7           5         29%    t1 is closed  yes
memo, writes ignored  7           4         43%    t1 is open    NO
sdkgen cache feature  7           4         43%    t1 is open    NO
HTTP cache (undici)   7           5         29%    t1 is closed  yes

The two highest savings come with the wrong answer. Both served the confirming read from an entry made before the write, so the agent reported t1 is open about a ticket it had just closed. A dashboard of hit rates alone would have ranked those two first.

The memo and the HTTP cache saved the same two customer reads by different routes. The memo cleared itself on the write. The HTTP cache sent a conditional request for the ticket, which the server marks no-cache. Its handler in undici 7.30.1 also drops a stored URL when an unsafe method against it succeeds, as RFC 9111 requires.

The Python twin has the same shape, without the in-flight handling a synchronous loop does not need. Python’s functools.lru_cache counts hits and misses through cache_info(), but it has no lifetime of its own and cache_clear() empties it for every tenant at once, so the twin keeps a dictionary.

    def key(self, tenant, tool, args):
        parts = [tenant, tool] if self.with_tenant else [tool]
        return json.dumps(parts + [args], sort_keys=True)

    def through(self, tenant, tool, args, call):
        if not tenant:
            raise ValueError("a cached tool call needs the caller identity")
        key = self.key(tenant, tool, args)
        entry = self.store.get(key)
        if entry and entry["expires"] > self.clock():
            self.hits += 1
            return copy.deepcopy(entry["value"])
        self.misses += 1
        value = call()
        self.store[key] = {"value": copy.deepcopy(value), "expires": self.clock() + self.ttl, "tenant": tenant}
        return value

python3 memo.py

answer: t1 is closed; api calls 4; hits 2, misses 3, hit rate 40%
no tenant in the key: acme sees Acme Rail, globex sees Acme Rail
tenant in the key: acme sees Acme Rail, globex sees Globex

Check it worked

Fourteen tests pin the measurements and the edges: expiry on an injected clock, argument order, parallel calls, and a read racing a write. The write may come through the same instance or another on the same store. They also cover a failed write that still clears the memo, the tenant taken from the client, and the sdkgen feature’s 5000 ms default.

node --test cache.test.mjs

1..14
# tests 14
# suites 0
# pass 14
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 421.183628

When it goes wrong

Globex’s agent reads Acme’s customer. The cache key has no tenant, takes it from somewhere other than the request, or the tenant travels in a header the cache never reads. Put the tenant the client sends first in every key.

node pitfall.mjs

two reads of t1, served with Cache-Control: no-cache, an ETag and Last-Modified
  sdkgen cache feature  requests 1
  HTTP cache (undici)   requests 2
  304 Not Modified answers sent: 1
customer c1 for acme, then for globex, through one cache
  memo keyed by tool and arguments            acme: Acme Rail, globex: Acme Rail
  memo keyed by tenant, tool and arguments    acme: Acme Rail, globex: Globex
  sdkgen cache, one client, tenant per call   acme: Acme Rail, globex: Acme Rail
  sdkgen cache, one client per tenant         acme: Acme Rail, globex: Globex
  HTTP cache shared, server sends Vary        acme: Acme Rail, globex: Globex

The sdkgen cache keys on method and URL, so one client that sends the tenant per call serves the first tenant’s record to the second. Each client holds its own store, so a client per tenant is safe. Stripe’s Connect API works this way, with the account in a header on each request.¹

The server says no-cache and the cache serves the record anyway. The sdkgen feature reads no caching headers, so its second read never reached the server. undici sent a conditional request and got 304 Not Modified. It stored that response only once the server added Last-Modified.²

The success rate drops after you add the cache. A write made the cached read stale. Clear the tenant’s entries on every write tool, or shorten the lifetime below the gap between a write and the read that checks it.

When not to do this

Do not turn on sdkgen’s cache feature for an agent that reads what it has just written. In 4.34.7 its lifetime, 5000 ms unless you set ttl, is the only thing that expires an entry. A write through the same client clears nothing, which the measurement on this page shows as a wrong answer. It suits reference data that changes less often than the lifetime.

Do not share a cache across tenants unless every key carries the tenant. In Redis that means the key string itself, set with an EX lifetime, because a shared store serves every caller.

Do not cache the result of a write, an error, or a read made while a write was in flight. None of them describes the state the next step will see.

Do not report a hit rate without the task success beside it.

Related how-tos

• Build cache keys that include the caller and Vary headers

• Share an API response cache across many workers with Redis

• Cut agent cost with provider prompt caching

• Add a dry-run mode to an agent’s write tools

Last verified

Verified 2026-10-15 against Node 22.22.0, Python 3.13.16, undici 7.30.1, @voxgig/create-sdkgen 0.30.7 and @voxgig/sdkgen 4.34.7. Every output block is what the command preceding it printed. The generated SDK’s compiled JavaScript is committed under generated/, and README.md beside it names the commands that produced it. Redis was not run; its row rests on its documentation.