# How to acknowledge a webhook fast and process it from a queue > Verify the signature, commit the delivery to durable storage, answer 200, and leave the slow work to a worker, so a sender's timeout cannot multiply one event. Source: https://voxgig.com/howto/acknowledge-webhooks-fast-and-process-from-a-queue - Audience: platform-team - Level: intermediate - Languages: typescript - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Verify the signature over the raw body, insert the delivery into durable storage keyed by its webhook-id, and answer 200 only after that write commits. A worker then claims stored rows, does the slow work, and marks each one done. Answer 503 when the write fails, so the sender keeps the delivery and retries. Verify before you store, or anyone can fill your queue. --- ## You will need Node 22.18 or later, which runs TypeScript files directly and ships [`node:sqlite`](https://nodejs.org/api/sqlite.html). A sender that signs its deliveries: the sample uses the `standardwebhooks` package against the [Standard Webhooks specification](https://github.com/standard-webhooks/standard-webhooks/blob/main/spec/standard-webhooks.md), which tells consumers to use the `webhook-id` header as an idempotency key. Verifying over the raw bytes is the subject of the how-to on reading the raw body for webhook verification in Node. Only the BullMQ variant needs a Redis server. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Amazon SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) | You run on AWS and want the queue's durability to be somebody else's pager | A 1 MiB message cap, so large payloads need a pointer to storage you also run | Your receiver is not on AWS, and a second cloud for one queue is overhead | | [BullMQ](https://docs.bullmq.io/guide/jobs/job-ids) | Node services that already run Redis and want retries, backoff, and concurrency per queue | Redis persistence you must configure, and job ids that reject a colon or digits alone | Redis would be infrastructure you add for this alone | | [Cloudflare Queues](https://developers.cloudflare.com/queues/configuration/javascript-apis/) | The receiver is a Worker, and `send()` resolving means the message is on disk | A 128 KB message cap, so large bodies go to R2 or D1 and the queue carries an id | The receiver runs outside Cloudflare | | [Inbox table in your database](https://www.sqlite.org/lang_returning.html) | One database you already back up, at volumes a table absorbs | You write the claim, the lease, and the retry schedule yourself, and the worker polls | Throughput where polling one table becomes the bottleneck | | [Inline processing](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks) | The work finishes well inside the sender's timeout, every time | Your slowest dependency sets your response time, and each timeout becomes a retry | The handler calls anything that can be slow or down | The sample below uses the inbox table, because it runs anywhere and every line of it is visible. SQS replaces the claim and lease code with its [visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html), and costs you a second store once payloads outgrow its message cap. BullMQ adds retries and concurrency limits without extra code, but its durability is whatever Redis is configured to write to disk. Inline processing costs nothing to build, and GitHub's own guidance on its ten-second limit is to queue the work instead. ## Verify, store, then answer The receiver does three things in a fixed order, and the order is the whole design. ```ts title="receiver.ts" export function createReceiver({ secret, inbox, maxBytes = 1024 * 1024, log = () => {} }: Options) { const wh = new Webhook(secret) return async function receive(req: IncomingMessage, res: ServerResponse): Promise { let body: Buffer try { body = await readBody(req, maxBytes) } catch { return reply(res, 413, { connection: 'close' }) } const headers = { 'webhook-id': one(req.headers['webhook-id']), 'webhook-timestamp': one(req.headers['webhook-timestamp']), 'webhook-signature': one(req.headers['webhook-signature']), } // 1. Verify before anything is stored. An unsigned request never reaches the inbox. try { wh.verify(body, headers, { jsonParse: false }) } catch (err) { log(`rejected: ${(err as Error).message}`) return reply(res, 401) } // 2. Store, and wait for the write to commit. try { await inbox.put({ id: headers['webhook-id'], receivedAt: Date.now(), body: body.toString('utf8') }) } catch (err) { // Nothing is stored, so say so. A 5xx makes the sender try again later. log(`store failed: ${(err as Error).message}`) return reply(res, 503) } // 3. Only now acknowledge. A duplicate gets a 200 too, so the sender stops retrying it. return reply(res, 200) } } ``` Verification comes first, so an unsigned request costs you a hash and nothing else. The store comes second, and the handler waits for it. A 200 tells the sender the delivery is safe, and a process that dies after a premature 200 has made that promise falsely. A duplicate gets a 200, because the sender is asking whether you have it, and you do. The 503 on a failed write matters as much as the 200 on a good one. It hands the delivery back to the sender's retry schedule, which exists for exactly this. ## Make the store the queue The inbox is one table. The primary key on `webhook-id` absorbs the sender's retries, and a claim with a lease lets several workers share it. ```ts title="inbox.ts" // One INSERT, committed before this returns. The primary key on the // webhook-id turns the sender's retry of a delivery you already hold into a no-op. async put(d: Delivery): Promise { this.db .prepare('INSERT OR IGNORE INTO webhook_inbox (id, received_at, body) VALUES (?, ?, ?)') .run(d.id, d.receivedAt, d.body) } // Claim the oldest row that is due. A claimed row gets a lease: if the worker // dies holding it, the row becomes due again when the lease runs out. claim(now: number, leaseMs: number): Claimed | undefined { const row = this.db.prepare(` UPDATE webhook_inbox SET status = 'working', attempts = attempts + 1, not_before = ? WHERE id = (SELECT id FROM webhook_inbox WHERE status IN ('pending', 'working') AND not_before <= ? ORDER BY received_at LIMIT 1) RETURNING id, received_at, body, attempts`).get(now + leaseMs, now) as { id: string; received_at: number; body: string; attempts: number } | undefined return row && { id: row.id, receivedAt: row.received_at, body: row.body, attempts: row.attempts } } ``` The claim is one `UPDATE ... RETURNING` statement, which SQLite runs atomically, so two workers cannot take the same row. `not_before` does two jobs: it is the lease while a row is being worked, and the retry delay after it fails. A row whose worker died becomes due again when its lease runs out. The constructor sets SQLite's `synchronous` level to EXTRA, which also syncs the journal's directory, so the row is on disk before the 200 goes out. The default, FULL, can lose the last commit to a power cut on some filesystems.[^2] ## Drain the inbox from a worker The worker reads the stored payload, calls your handler, and marks the row done. ```ts title="worker.ts" // Process every row that is due, one at a time, and return how many were handled. export async function drain(inbox: SqliteInbox, handle: Handler, { leaseMs = 60_000, maxAttempts = 8, retryInMs = (attempts) => 1000 * 2 ** attempts, now = Date.now, }: WorkerOptions = {}): Promise { let handled = 0 for (let job = inbox.claim(now(), leaseMs); job; job = inbox.claim(now(), leaseMs)) { handled++ if (job.attempts > maxAttempts) { // Every earlier claim ended with its worker dying, so the handler never reported back. inbox.fail(job, `no worker finished it in ${maxAttempts} attempts`, now(), 0, 0) continue } let event: unknown try { event = JSON.parse(job.body) } catch { // A body that does not parse will not parse on the next attempt either. inbox.fail(job, 'body is not JSON', now(), 0, 0) continue } try { await handle(event, { id: job.id, attempts: job.attempts }) inbox.complete(job) } catch (err) { inbox.fail(job, String((err as Error)?.message ?? err), now(), retryInMs(job.attempts), maxAttempts) } } return handled } ``` A failing handler puts the row back with an exponential delay and buries it after eight attempts. A body that is not JSON is buried at once, since a retry cannot fix it, and so is a row whose earlier workers all died before they reported back. `complete` and `fail` match on the attempt count the claim returned, so a worker whose lease ran out cannot overwrite the outcome of the worker that took over. ## Watch one slow event under Slack's timing The demo sends one event with Slack's policy scaled down: a 300 ms timeout instead of three seconds, and three retries tens of milliseconds apart instead of minutes. The CRM call takes 600 ms. ```bash node --no-warnings demo.ts ``` ```text output inline: verify, update the CRM, then answer attempt 1 no answer within 300 ms attempt 2 no answer within 300 ms attempt 3 no answer within 300 ms attempt 4 no answer within 300 ms CRM updates for one event: 4 queue-first: verify, store, answer; a worker updates the CRM attempt 1 200 inbox before the worker runs: {"pending":1} inbox after the worker runs: {"done":1} CRM updates for one event: 1 ``` The inline handler never answered in time, so the sender tried four times and the CRM was updated four times for one deal. Each retry also started while the previous update was still running, which is how a slow dependency becomes a load spike. The queue-first receiver answered the first attempt, and the worker made one call. `--no-warnings` hides the experimental notice `node:sqlite` prints. ## Put BullMQ behind the same receiver The receiver depends only on `put()`, so a BullMQ queue can replace the table without touching it. ```ts title="bullmq-inbox.ts" // BullMQ refuses a custom job id that contains ':' or is all digits, and a // sender's webhook-id can be either. Hash it into a shape BullMQ always accepts. export function jobIdFor(webhookId: string): string { return 'wh-' + createHash('sha256').update(webhookId).digest('hex').slice(0, 40) } export const jobOptions: JobsOptions = { attempts: 8, backoff: { type: 'exponential', delay: 1000 }, // A removed job no longer blocks a duplicate id, so keep finished jobs for // longer than any sender keeps retrying. Stripe retries for three days. removeOnComplete: { age: 4 * 24 * 3600 }, removeOnFail: { age: 14 * 24 * 3600 }, } export class BullmqInbox implements Inbox { queue: Queue constructor(connection: ConnectionOptions, name = 'webhooks') { this.queue = new Queue(name, { connection }) } // Resolves once Redis has the job. How much survives a Redis crash after that // depends on Redis's own persistence settings, not on this call. async put(d: Delivery): Promise { await this.queue.add('webhook', d, { ...jobOptions, jobId: jobIdFor(d.id) }) } close(): Promise { return this.queue.close() } } ``` Two details in that file come from BullMQ's [job id documentation](https://docs.bullmq.io/guide/jobs/job-ids). Both held when BullMQ 6.3.9 ran against Redis 7.0.15, in the session that wrote this page and again in review. A custom job id with a colon, such as `evt:3`, or of digits alone, such as `123`, is refused with an error, so the webhook-id is hashed first. And a removed job no longer counts as a duplicate, so finished jobs are kept for longer than [Stripe's three days](https://docs.stripe.com/webhooks) of retries. `bullmq-demo.ts` runs the receiver against this inbox. It needs Redis, which the CI runner that re-checks this page does not have, so its output is not captured here. It was run in the session that wrote the page, against Redis 7.0.15. It answered four deliveries with 200, handled three, and left three completed jobs: the repeated `msg_2` was absorbed by its job id, and `evt:3` went through hashed. BullMQ 6.3.9 also needs `ioredis` installed beside it, and says so in an error if it is missing. `put()` resolving means Redis has the job in memory. Whether it survives a Redis crash depends on [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/), where `appendfsync everysec` can lose the last second of writes. BullMQ's [production guide](https://docs.bullmq.io/guide/going-to-production) recommends turning AOF on. ## Check it worked The suite covers the receiver, the inbox, and the worker. The test that guards the ordering is the failed store: a receiver that answers before writing would return 200 here and lose the delivery. A second test gives the store a 100 ms commit and checks that the row exists when the 200 arrives. ```ts title="receiver.test.ts" test('a store that fails answers 503, so the sender keeps the delivery and retries', async () => { const broken: Inbox = { put: async () => { throw new Error('disk full') } } assert.equal(await post(createReceiver({ secret: SECRET, inbox: broken }), signedHeaders('msg_1', body), body), 503) }) ``` ```bash node --test receiver.test.ts ``` ```text output 1..14 # tests 14 # suites 0 # pass 14 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 493.268537 ``` Fourteen tests pass. The others pin the duplicate, a bad signature, a stale timestamp, a body over the size limit, and the sync level. The worker's tests cover the lease, a late result from a worker that lost its lease, the retry delay, and the burial after the last attempt. ## When it goes wrong Deliveries the sender saw acknowledged never get processed. The receiver answered before the write was durable, then the process died. The first half of `pitfall.ts` sends three events to a receiver in a child process, then kills it with `SIGKILL` and starts it again on the same database. ```bash node --no-warnings pitfall.ts ``` ```text output 1. answer 200, then die before the work is done ack-first answered 200: 3 of 3 processed before the kill: 0 processed after a restart: 0 of 3 store-first answered 200: 3 of 3 processed before the kill: 0 processed after a restart: 3 of 3 2. store first, verify later, and take 200 unsigned requests store-then-verify answered 200 requests with 200 inbox: 200 rows, 2000 KB verify-then-store answered 200 requests with 401 inbox: 0 rows, 0 KB ``` Both receivers answered 200 three times, so the sender counts all three as delivered and will not send them again. The one that kept its work in memory lost all three when its process died. The second half shows the other ordering mistake: storing before verifying let 200 unsigned requests put 2 MB in the inbox. A delivery is processed twice although the sender sent it once. The lease ran out while the handler was still working, so a second worker claimed the row. Set the lease longer than your handler's own timeout, and make the handler idempotent on the `webhook-id`. Pending rows pile up without an error in the log. The worker is not running, or it is stuck on one row. Alert on the age of the oldest pending row, not on the count, because a slow week and a dead worker can show the same count. ## When not to do this Do not add a queue to a handler that answers in a few milliseconds and calls nothing that can be slow. Logging an event or flipping a flag inline is cheaper than a table, a worker, and a lease to operate. Do not queue before verifying, even to keep the receiver thin. The pitfall run's 200 unsigned requests put 2 MB in the inbox, and the inbox grows by each body an attacker sends after them. Do not put the whole body on SQS or Cloudflare Queues when payloads can be large. GitHub caps a payload at 25 MB, far over both message limits, so store the body and queue its id.[^3] Do not treat the worker as exactly once. The sender retries, the lease expires, and BullMQ [moves a stalled job back to waiting](https://docs.bullmq.io/guide/jobs/stalled), so the handler sees some deliveries twice. ## Related how-tos - [Verify a Standard Webhooks signature in PHP](/howto/verify-a-standard-webhooks-signature-in-php) - [Make queue consumer writes safe under redelivery](/howto/queue-consumer-writes-under-at-least-once) - [Limit concurrency in a message worker](/howto/limit-concurrency-in-a-message-worker) - [Build a queue-backed webhook sender on BullMQ](/howto/build-a-queue-backed-webhook-sender-on-bullmq) ## Last verified Verified 2026-10-15 against Node 22.22.0 and 22.23.3, standardwebhooks 1.1.1, BullMQ 6.3.9, and `ioredis` 5.11.1. Every output block is what the command before it printed, on both Node versions. The BullMQ run used Redis 7.0.15 in the writing session and is described rather than captured. [^1]: Slack's documentation announces its retries as three knocks on your server's door, one per retry, and then gives the schedule: [nearly immediately, after one minute, and after five minutes](https://docs.slack.dev/apis/events-api/). Each retry carries an `x-slack-retry-num` header, so a handler that is counting can count along. [^2]: SQLite's documentation is more careful than most about the last letter of ACID. With a rollback journal, the default level, FULL, is described as atomic, consistent, and isolated, and as something that ["might also be durable"](https://www.sqlite.org/pragma.html#pragma_synchronous), depending on the filesystem. The next level up, EXTRA, exists to sync the directory once the journal is deleted, for the commit that is followed closely by a power cut. [^3]: Past the cap, [GitHub delivers no payload](https://docs.github.com/en/webhooks/webhook-events-and-payloads) for the event at all rather than a truncated one. A body at the cap would fill about two hundred Cloudflare Queues messages at 128 KB each, or twenty-four on SQS.