How to make queue consumer writes safe under redelivery#

A deploy restarts a consumer mid-batch, the broker redelivers what was not acknowledged, and two customers are charged twice. The handler was correct, the API call succeeded both times, and the only thing that went wrong is that the same message arrived twice, which the broker documented from the beginning.

What you get

You will end up with a consumer whose writes are safe to repeat, because the key it sends comes from the message rather than from the attempt. This is for you if a worker in your system calls an API that changes something.

Short answer

Send an idempotency key derived from the message id on every write the consumer makes. A redelivery then replays the first response instead of creating a second charge. Add a seen set in the consumer to skip the call entirely. Treat that as an optimization rather than as the control, because a restart leaves the set empty.

You will need

Node 22 or later, and a consumer that makes writes against an API. The key travels in an Idempotency-Key header, as the IETF draft describes and as Stripe implemented before it.¹

Approaches compared

The last row is the one worth reading twice. A broker’s exactly once semantics cover what the broker can see. The moment your handler makes an HTTP call to somebody else, the delivery guarantee ends and the outcome depends on whether that API deduplicates.

The seen set and the key are not alternatives either. The set saves a network call, and the key is what holds when the set is gone. Running both is normal, but only the key is the control.

Derive the key from the message

The key has to be the same on every attempt, which means it comes from the message.

export const keyed = (api) => (message) =>
  api.charge({ amount: message.amount, key: `msg:${message.id}` })

Generating a key per attempt is the mistake that looks correct. A fresh UUID per call means every redelivery presents a key the API has never seen, and the API does exactly what it was asked to do.

Prefix the key with something that says where it came from. msg: in a shared key space keeps a message id from colliding with an order id somebody else derived a key from.

Treat the local set as a shortcut

The set is fast and it is not the guarantee.

export function deduped(api, seen = new Map()) {
  return (message) => {
    if (seen.has(message.id)) return { ...seen.get(message.id), skipped: true }
    const result = api.charge({ amount: message.amount, key: `msg:${message.id}` })
    seen.set(message.id, result)
    return result
  }
}

It still sends the key. A consumer that relies on the set alone is correct until the first restart, the first scale-out to two instances, or the first time the map is cleared to save memory.

Bound the set. An unbounded map of every message id the process has seen is a leak with a delay on it, and a window of a few minutes covers the redeliveries that actually happen. Anything older has been through a dead letter queue, and that deserves attention rather than a silent skip.

Check it worked

Deliver three messages with two of them redelivered.

node demo.mjs

naive                5 deliveries  5 charges  total 22650  0 replayed  0 skipped before the call
idempotency key      5 deliveries  3 charges  total 11550  2 replayed  0 skipped before the call
key and a seen set   5 deliveries  3 charges  total 11550  0 replayed  2 skipped before the call

what the naive consumer charged: ch_1=1200 ch_2=1200 ch_3=450 ch_4=9900 ch_5=9900

The totals are the finding. Five deliveries produced 22,650 with a naive consumer and 11,550 with either of the other two. The last line names the duplicates: two charges of 1,200 and two of 9,900, for one message each.

Rows two and three reach the same state by different routes. One lets the call go and the API refuses to repeat the work; the other never makes the second call. Both are safe until a restart, after which only the first stays safe.

The replayed and skipped counts are worth keeping as metrics. A rising replay count means the broker is redelivering more than it used to, which is usually a consumer acknowledging too late, and that is a different problem with the same symptom.

node --test consumer.test.mjs

1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 124.575952

When it goes wrong

Duplicates appear after a deploy and not before. The consumer relied on an in-process set. Send the key, and keep the set as the optimization it is.

The API returns a conflict rather than a replay. Two different payloads were sent under one key, which the API is right to refuse. Include the fields that define the write in the key derivation, or fix whatever is changing the payload between attempts. A timestamp inside the body is the usual culprit, and it makes every attempt a different request.

Everything works, but the ledger is still wrong. The duplicate is upstream: two messages were published for one event. Key on the event rather than on the message, if the publisher gives you one. An event id that survives a republish is the thing to ask the publisher for, and most of them already have one internally.

The key expires before the redelivery. The API remembers keys for a window and the message sat in a dead letter queue for a day. Check the window, and treat a replay outside it as a new write that needs a human. A dead letter queue and an idempotency window are rarely sized against each other, so nobody notices until they disagree.

When not to do this

Do not send an idempotency key on a read. It costs the API storage, buys nothing, and a cached read served from a key store is a bug waiting for a stale value.

Do not derive the key from the payload alone. Two genuinely separate charges for the same amount to the same customer are a real thing, and a key made from the fields would merge them.²

Do not rely on the broker’s exactly once mode for an external write. It describes the broker’s own guarantees. Your handler’s HTTP call is outside them, and the setting will not say so.

Related how-tos

• Limit concurrency in a message worker

• Retry fetch calls with exponential backoff in Node.js

Last verified

Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.