# How to make queue consumer writes safe under redelivery > Derive an idempotency key from the message rather than from the attempt, so a broker that delivers at least once cannot charge a customer twice. Source: https://voxgig.com/howto/queue-consumer-writes-under-at-least-once - Audience: platform-team - Level: intermediate - Languages: typescript, javascript - Verified: 2026-09-14 - Published: 2026-09-14 - Updated: 2026-09-24 ## Short answer Send an idempotency key derived from the message id on every write the consumer makes. A redelivery then replays the first response instead of creating a second charge. Add a seen set in the consumer to skip the call entirely. Treat that as an optimization rather than as the control, because a restart leaves the set empty. --- ## You will need Node 22 or later, and a consumer that makes writes against an API. The key travels in an `Idempotency-Key` header, as [the IETF draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) describes and as [Stripe](https://docs.stripe.com/api/idempotent_requests) implemented before it.[^1] ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A seen set in the consumer](https://nodejs.org/api/globals.html#class-map) | Cutting the cost of obvious duplicates inside one process | Nothing survives a restart, so it is an optimization rather than a guarantee | The duplicate must be stopped even after a deploy | | [A unique constraint in the database](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) | Writes you own, where the duplicate can be rejected by the store | An error path to handle, and nothing if the write goes to somebody else's API | The write is a call to an API you do not run | | [An idempotency key from the message id](https://docs.stripe.com/api/idempotent_requests) | Any write against an API that supports the header, which is most payment APIs | A key the API remembers for a window, and a key derivation you must get right | The API ignores the header entirely | | [Exactly once delivery from the broker](https://kafka.apache.org/documentation/#semantics) | Work that stays inside one system with transactional support | A strong guarantee inside the broker that stops at the edge of your API call | The consumer calls an external API, which no broker can enrol in its transaction | The last row is the one worth reading twice. A broker's exactly once semantics cover what the broker can see. The moment your handler makes an HTTP call to somebody else, the delivery guarantee ends and the outcome depends on whether that API deduplicates. The seen set and the key are not alternatives either. The set saves a network call, and the key is what holds when the set is gone. Running both is normal, but only the key is the control. ## Derive the key from the message The key has to be the same on every attempt, which means it comes from the message. ```js title="consumer.mjs" export const keyed = (api) => (message) => api.charge({ amount: message.amount, key: `msg:${message.id}` }) ``` Generating a key per attempt is the mistake that looks correct. A fresh UUID per call means every redelivery presents a key the API has never seen, and the API does exactly what it was asked to do. Prefix the key with something that says where it came from. `msg:` in a shared key space keeps a message id from colliding with an order id somebody else derived a key from. ## Treat the local set as a shortcut The set is fast and it is not the guarantee. ```js title="consumer.mjs" export function deduped(api, seen = new Map()) { return (message) => { if (seen.has(message.id)) return { ...seen.get(message.id), skipped: true } const result = api.charge({ amount: message.amount, key: `msg:${message.id}` }) seen.set(message.id, result) return result } } ``` It still sends the key. A consumer that relies on the set alone is correct until the first restart, the first scale-out to two instances, or the first time the map is cleared to save memory. Bound the set. An unbounded map of every message id the process has seen is a leak with a delay on it, and a window of a few minutes covers the redeliveries that actually happen. Anything older has been through a dead letter queue, and that deserves attention rather than a silent skip. ## Check it worked Deliver three messages with two of them redelivered. ```bash node demo.mjs ``` ```text output naive 5 deliveries 5 charges total 22650 0 replayed 0 skipped before the call idempotency key 5 deliveries 3 charges total 11550 2 replayed 0 skipped before the call key and a seen set 5 deliveries 3 charges total 11550 0 replayed 2 skipped before the call what the naive consumer charged: ch_1=1200 ch_2=1200 ch_3=450 ch_4=9900 ch_5=9900 ``` The totals are the finding. Five deliveries produced 22,650 with a naive consumer and 11,550 with either of the other two. The last line names the duplicates: two charges of 1,200 and two of 9,900, for one message each. Rows two and three reach the same state by different routes. One lets the call go and the API refuses to repeat the work; the other never makes the second call. Both are safe until a restart, after which only the first stays safe. The replayed and skipped counts are worth keeping as metrics. A rising replay count means the broker is redelivering more than it used to, which is usually a consumer acknowledging too late, and that is a different problem with the same symptom. ```bash node --test consumer.test.mjs ``` ```text output 1..6 # tests 6 # suites 0 # pass 6 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 124.575952 ``` ## When it goes wrong Duplicates appear after a deploy and not before. The consumer relied on an in-process set. Send the key, and keep the set as the optimization it is. The API returns a conflict rather than a replay. Two different payloads were sent under one key, which the API is right to refuse. Include the fields that define the write in the key derivation, or fix whatever is changing the payload between attempts. A timestamp inside the body is the usual culprit, and it makes every attempt a different request. Everything works, but the ledger is still wrong. The duplicate is upstream: two messages were published for one event. Key on the event rather than on the message, if the publisher gives you one. An event id that survives a republish is the thing to ask the publisher for, and most of them already have one internally. The key expires before the redelivery. The API remembers keys for a window and the message sat in a dead letter queue for a day. Check the window, and treat a replay outside it as a new write that needs a human. A dead letter queue and an idempotency window are rarely sized against each other, so nobody notices until they disagree. ## When not to do this Do not send an idempotency key on a read. It costs the API storage, buys nothing, and a cached read served from a key store is a bug waiting for a stale value. Do not derive the key from the payload alone. Two genuinely separate charges for the same amount to the same customer are a real thing, and a key made from the fields would merge them.[^2] Do not rely on the broker's exactly once mode for an external write. It describes the broker's own guarantees. Your handler's HTTP call is outside them, and the setting will not say so. ## Related how-tos - [Limit concurrency in a message worker](/howto/limit-concurrency-in-a-message-worker) - [Retry fetch calls with exponential backoff in Node.js](/howto/retry-fetch-with-backoff-in-node) ## Last verified Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed. [^1]: Stripe allows [a key of up to 255 characters](https://docs.stripe.com/api/idempotent_requests) and suggests a version 4 UUID, whose text form [RFC 9562](https://www.rfc-editor.org/rfc/rfc9562.html#section-4) fixes at 32 hexadecimal digits and four hyphens. A key of `msg:` and a UUID spends 40 of the 255, and the remaining 215 are available to anyone with a longer opinion about where the message came from. [^2]: A unique constraint has the opposite blind spot. [PostgreSQL](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) does not treat two nulls as equal, so a table may hold any number of rows whose constrained column is null. The documentation records that the SQL standard leaves the choice to the implementation. [PostgreSQL 15](https://www.postgresql.org/docs/release/15.0/) added `NULLS NOT DISTINCT` in October 2022, for anyone who wanted the constraint to mean what its name says.