How to acknowledge a webhook fast and process it from a queue#

Your webhook handler updates the CRM before it answers, and the CRM takes longer than the three seconds Slack waits. Slack marks the attempt failed and sends the event again while the first update is still running.¹ One event becomes four CRM calls. When the CRM is down, every event arrives four times at a handler that cannot succeed.

What you get

You will end up with a receiver that answers in milliseconds, an inbox that survives the process dying, and a worker that retries on its own schedule. This is for you if a sender’s timeout decides how often your handler runs.

Short answer

Verify the signature over the raw body, insert the delivery into durable storage keyed by its webhook-id, and answer 200 only after that write commits. A worker then claims stored rows, does the slow work, and marks each one done. Answer 503 when the write fails, so the sender keeps the delivery and retries. Verify before you store, or anyone can fill your queue.

You will need

Node 22.18 or later, which runs TypeScript files directly and ships node:sqlite. A sender that signs its deliveries: the sample uses the standardwebhooks package against the Standard Webhooks specification, which tells consumers to use the webhook-id header as an idempotency key. Verifying over the raw bytes is the subject of the how-to on reading the raw body for webhook verification in Node. Only the BullMQ variant needs a Redis server.

Approaches compared

The sample below uses the inbox table, because it runs anywhere and every line of it is visible. SQS replaces the claim and lease code with its visibility timeout, and costs you a second store once payloads outgrow its message cap. BullMQ adds retries and concurrency limits without extra code, but its durability is whatever Redis is configured to write to disk. Inline processing costs nothing to build, and GitHub’s own guidance on its ten-second limit is to queue the work instead.

Verify, store, then answer

The receiver does three things in a fixed order, and the order is the whole design.

export function createReceiver({ secret, inbox, maxBytes = 1024 * 1024, log = () => {} }: Options) {
  const wh = new Webhook(secret)

  return async function receive(req: IncomingMessage, res: ServerResponse): Promise<void> {
    let body: Buffer
    try {
      body = await readBody(req, maxBytes)
    } catch {
      return reply(res, 413, { connection: 'close' })
    }

    const headers = {
      'webhook-id': one(req.headers['webhook-id']),
      'webhook-timestamp': one(req.headers['webhook-timestamp']),
      'webhook-signature': one(req.headers['webhook-signature']),
    }

    // 1. Verify before anything is stored. An unsigned request never reaches the inbox.
    try {
      wh.verify(body, headers, { jsonParse: false })
    } catch (err) {
      log(`rejected: ${(err as Error).message}`)
      return reply(res, 401)
    }

    // 2. Store, and wait for the write to commit.
    try {
      await inbox.put({ id: headers['webhook-id'], receivedAt: Date.now(), body: body.toString('utf8') })
    } catch (err) {
      // Nothing is stored, so say so. A 5xx makes the sender try again later.
      log(`store failed: ${(err as Error).message}`)
      return reply(res, 503)
    }

    // 3. Only now acknowledge. A duplicate gets a 200 too, so the sender stops retrying it.
    return reply(res, 200)
  }
}

Verification comes first, so an unsigned request costs you a hash and nothing else. The store comes second, and the handler waits for it. A 200 tells the sender the delivery is safe, and a process that dies after a premature 200 has made that promise falsely. A duplicate gets a 200, because the sender is asking whether you have it, and you do.

The 503 on a failed write matters as much as the 200 on a good one. It hands the delivery back to the sender’s retry schedule, which exists for exactly this.

Make the store the queue

The inbox is one table. The primary key on webhook-id absorbs the sender’s retries, and a claim with a lease lets several workers share it.

  // One INSERT, committed before this returns. The primary key on the
  // webhook-id turns the sender's retry of a delivery you already hold into a no-op.
  async put(d: Delivery): Promise<void> {
    this.db
      .prepare('INSERT OR IGNORE INTO webhook_inbox (id, received_at, body) VALUES (?, ?, ?)')
      .run(d.id, d.receivedAt, d.body)
  }

  // Claim the oldest row that is due. A claimed row gets a lease: if the worker
  // dies holding it, the row becomes due again when the lease runs out.
  claim(now: number, leaseMs: number): Claimed | undefined {
    const row = this.db.prepare(`
      UPDATE webhook_inbox
         SET status = 'working', attempts = attempts + 1, not_before = ?
       WHERE id = (SELECT id FROM webhook_inbox
                    WHERE status IN ('pending', 'working') AND not_before <= ?
                    ORDER BY received_at LIMIT 1)
      RETURNING id, received_at, body, attempts`).get(now + leaseMs, now) as
      { id: string; received_at: number; body: string; attempts: number } | undefined
    return row && { id: row.id, receivedAt: row.received_at, body: row.body, attempts: row.attempts }
  }

The claim is one UPDATE ... RETURNING statement, which SQLite runs atomically, so two workers cannot take the same row. not_before does two jobs: it is the lease while a row is being worked, and the retry delay after it fails. A row whose worker died becomes due again when its lease runs out. The constructor sets SQLite’s synchronous level to EXTRA, which also syncs the journal’s directory, so the row is on disk before the 200 goes out. The default, FULL, can lose the last commit to a power cut on some filesystems.²

Drain the inbox from a worker

The worker reads the stored payload, calls your handler, and marks the row done.

// Process every row that is due, one at a time, and return how many were handled.
export async function drain(inbox: SqliteInbox, handle: Handler, {
  leaseMs = 60_000,
  maxAttempts = 8,
  retryInMs = (attempts) => 1000 * 2 ** attempts,
  now = Date.now,
}: WorkerOptions = {}): Promise<number> {
  let handled = 0
  for (let job = inbox.claim(now(), leaseMs); job; job = inbox.claim(now(), leaseMs)) {
    handled++
    if (job.attempts > maxAttempts) {
      // Every earlier claim ended with its worker dying, so the handler never reported back.
      inbox.fail(job, `no worker finished it in ${maxAttempts} attempts`, now(), 0, 0)
      continue
    }
    let event: unknown
    try {
      event = JSON.parse(job.body)
    } catch {
      // A body that does not parse will not parse on the next attempt either.
      inbox.fail(job, 'body is not JSON', now(), 0, 0)
      continue
    }
    try {
      await handle(event, { id: job.id, attempts: job.attempts })
      inbox.complete(job)
    } catch (err) {
      inbox.fail(job, String((err as Error)?.message ?? err), now(), retryInMs(job.attempts), maxAttempts)
    }
  }
  return handled
}

A failing handler puts the row back with an exponential delay and buries it after eight attempts. A body that is not JSON is buried at once, since a retry cannot fix it, and so is a row whose earlier workers all died before they reported back. complete and fail match on the attempt count the claim returned, so a worker whose lease ran out cannot overwrite the outcome of the worker that took over.

Watch one slow event under Slack’s timing

The demo sends one event with Slack’s policy scaled down: a 300 ms timeout instead of three seconds, and three retries tens of milliseconds apart instead of minutes. The CRM call takes 600 ms.

node --no-warnings demo.ts

inline: verify, update the CRM, then answer
  attempt 1  no answer within 300 ms
  attempt 2  no answer within 300 ms
  attempt 3  no answer within 300 ms
  attempt 4  no answer within 300 ms
  CRM updates for one event: 4

queue-first: verify, store, answer; a worker updates the CRM
  attempt 1  200
  inbox before the worker runs: {"pending":1}
  inbox after the worker runs:  {"done":1}
  CRM updates for one event: 1

The inline handler never answered in time, so the sender tried four times and the CRM was updated four times for one deal. Each retry also started while the previous update was still running, which is how a slow dependency becomes a load spike. The queue-first receiver answered the first attempt, and the worker made one call. --no-warnings hides the experimental notice node:sqlite prints.

Put BullMQ behind the same receiver

The receiver depends only on put(), so a BullMQ queue can replace the table without touching it.

// BullMQ refuses a custom job id that contains ':' or is all digits, and a
// sender's webhook-id can be either. Hash it into a shape BullMQ always accepts.
export function jobIdFor(webhookId: string): string {
  return 'wh-' + createHash('sha256').update(webhookId).digest('hex').slice(0, 40)
}

export const jobOptions: JobsOptions = {
  attempts: 8,
  backoff: { type: 'exponential', delay: 1000 },
  // A removed job no longer blocks a duplicate id, so keep finished jobs for
  // longer than any sender keeps retrying. Stripe retries for three days.
  removeOnComplete: { age: 4 * 24 * 3600 },
  removeOnFail: { age: 14 * 24 * 3600 },
}

export class BullmqInbox implements Inbox {
  queue: Queue

  constructor(connection: ConnectionOptions, name = 'webhooks') {
    this.queue = new Queue(name, { connection })
  }

  // Resolves once Redis has the job. How much survives a Redis crash after that
  // depends on Redis's own persistence settings, not on this call.
  async put(d: Delivery): Promise<void> {
    await this.queue.add('webhook', d, { ...jobOptions, jobId: jobIdFor(d.id) })
  }

  close(): Promise<void> {
    return this.queue.close()
  }
}

Two details in that file come from BullMQ’s job id documentation. Both held when BullMQ 6.3.9 ran against Redis 7.0.15, in the session that wrote this page and again in review. A custom job id with a colon, such as evt:3, or of digits alone, such as 123, is refused with an error, so the webhook-id is hashed first. And a removed job no longer counts as a duplicate, so finished jobs are kept for longer than Stripe’s three days of retries.

bullmq-demo.ts runs the receiver against this inbox. It needs Redis, which the CI runner that re-checks this page does not have, so its output is not captured here. It was run in the session that wrote the page, against Redis 7.0.15. It answered four deliveries with 200, handled three, and left three completed jobs: the repeated msg_2 was absorbed by its job id, and evt:3 went through hashed. BullMQ 6.3.9 also needs ioredis installed beside it, and says so in an error if it is missing.

put() resolving means Redis has the job in memory. Whether it survives a Redis crash depends on Redis persistence, where appendfsync everysec can lose the last second of writes. BullMQ’s production guide recommends turning AOF on.

Check it worked

The suite covers the receiver, the inbox, and the worker. The test that guards the ordering is the failed store: a receiver that answers before writing would return 200 here and lose the delivery. A second test gives the store a 100 ms commit and checks that the row exists when the 200 arrives.

test('a store that fails answers 503, so the sender keeps the delivery and retries', async () => {
  const broken: Inbox = { put: async () => { throw new Error('disk full') } }
  assert.equal(await post(createReceiver({ secret: SECRET, inbox: broken }), signedHeaders('msg_1', body), body), 503)
})

node --test receiver.test.ts

1..14
# tests 14
# suites 0
# pass 14
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 493.268537

Fourteen tests pass. The others pin the duplicate, a bad signature, a stale timestamp, a body over the size limit, and the sync level. The worker’s tests cover the lease, a late result from a worker that lost its lease, the retry delay, and the burial after the last attempt.

When it goes wrong

Deliveries the sender saw acknowledged never get processed. The receiver answered before the write was durable, then the process died. The first half of pitfall.ts sends three events to a receiver in a child process, then kills it with SIGKILL and starts it again on the same database.

node --no-warnings pitfall.ts

1. answer 200, then die before the work is done

ack-first    answered 200: 3 of 3   processed before the kill: 0   processed after a restart: 0 of 3
store-first  answered 200: 3 of 3   processed before the kill: 0   processed after a restart: 3 of 3

2. store first, verify later, and take 200 unsigned requests

store-then-verify  answered 200 requests with 200   inbox: 200 rows, 2000 KB
verify-then-store  answered 200 requests with 401   inbox: 0 rows, 0 KB

Both receivers answered 200 three times, so the sender counts all three as delivered and will not send them again. The one that kept its work in memory lost all three when its process died. The second half shows the other ordering mistake: storing before verifying let 200 unsigned requests put 2 MB in the inbox.

A delivery is processed twice although the sender sent it once. The lease ran out while the handler was still working, so a second worker claimed the row. Set the lease longer than your handler’s own timeout, and make the handler idempotent on the webhook-id.

Pending rows pile up without an error in the log. The worker is not running, or it is stuck on one row. Alert on the age of the oldest pending row, not on the count, because a slow week and a dead worker can show the same count.

When not to do this

Do not add a queue to a handler that answers in a few milliseconds and calls nothing that can be slow. Logging an event or flipping a flag inline is cheaper than a table, a worker, and a lease to operate.

Do not queue before verifying, even to keep the receiver thin. The pitfall run’s 200 unsigned requests put 2 MB in the inbox, and the inbox grows by each body an attacker sends after them.

Do not put the whole body on SQS or Cloudflare Queues when payloads can be large. GitHub caps a payload at 25 MB, far over both message limits, so store the body and queue its id.³

Do not treat the worker as exactly once. The sender retries, the lease expires, and BullMQ moves a stalled job back to waiting, so the handler sees some deliveries twice.

Related how-tos

• Verify a Standard Webhooks signature in PHP

• Make queue consumer writes safe under redelivery

• Limit concurrency in a message worker

• Build a queue-backed webhook sender on BullMQ

Last verified

Verified 2026-10-15 against Node 22.22.0 and 22.23.3, standardwebhooks 1.1.1, BullMQ 6.3.9, and ioredis 5.11.1. Every output block is what the command before it printed, on both Node versions. The BullMQ run used Redis 7.0.15 in the writing session and is described rather than captured.