How to build a queue-backed webhook sender on BullMQ#

Your API posts each webhook from inside the request that created the event. One customer’s endpoint stops answering, so POST /orders waits for it until the HTTP client gives up, and a delivery that fails is gone because nothing retries it. Your API’s latency is set by the slowest server any customer runs.

What you get

You will end up with a sender that records each event and its deliveries, runs every POST from a BullMQ worker, and keeps the attempt log a customer will ask to see. This is for you if your API sends webhooks to endpoints you do not control.

Short answer

Write the event row and one delivery row per subscribed endpoint, then add one BullMQ job per delivery, with the delivery id as the job id. Workers POST with a ten-second timeout, record every attempt with its status and response, and retry on the Standard Webhooks schedule. Give each endpoint its own queue, so a slow customer holds only its own concurrency.

You will need

Node 22.18 or later, and Redis for BullMQ in production. The samples run without a server: with no REDIS_URL set, they use the PostgreSQL backend BullMQ ships on PGlite, Postgres compiled to WebAssembly and run in process. With REDIS_URL set, backend.ts hands every field of the URL to the client: the user, the password, the database number, and the TLS that rediss:// asks for. Signing follows the Standard Webhooks specification, which the how-to on signing outbound webhooks covers in detail.

Approaches compared

BullMQ gives you the concurrency cap and the retry schedule as configuration, and costs you Redis and a sweeper for deliveries whose queue write never happened. pg-boss removes the sweeper, because it can insert a job inside the transaction that wrote the event, and puts the queue’s write load on your database. Svix removes all of it and bills per message.

Record the event and its deliveries first

The request path writes rows, not HTTP. The event and one delivery per enabled endpoint go in together, so an event never exists without the deliveries that carry it.

  createEvent(id: string, type: string, payload: unknown, now = Date.now()): string[] {
    this.db.exec('BEGIN IMMEDIATE')
    try {
      const fresh = this.db.prepare('INSERT OR IGNORE INTO events (id, type, payload, created_at) VALUES (?, ?, ?, ?)')
        .run(id, type, JSON.stringify(payload), now).changes === 1
      const ids: string[] = []
      if (fresh) {
        // The delivery id is also the BullMQ job id, which may not contain a ':'. Both parts
        // are percent-encoded, '-' included, so the '-' that joins them is the only one, and
        // two different event and endpoint pairs never share an id.
        const part = (s: string) => encodeURIComponent(s).replaceAll('-', '%2D')
        for (const ep of this.db.prepare("SELECT id FROM endpoints WHERE status = 'enabled' ORDER BY id").all() as { id: string }[]) {
          const deliveryId = `${part(id)}-${part(ep.id)}`
          this.db.prepare('INSERT INTO deliveries (id, event_id, endpoint_id, created_at) VALUES (?, ?, ?, ?)')
            .run(deliveryId, id, ep.id, now)
          ids.push(deliveryId)
        }
      }
      this.db.exec('COMMIT')
      return ids
    } catch (err) {
      this.db.exec('ROLLBACK')
      throw err
    }
  }

The delivery id doubles as the BullMQ job id, so adding the same delivery twice adds one job. BullMQ 6.3.9 refuses a custom job id that contains a colon, on Redis and on PostgreSQL alike, so both parts of the id are percent-encoded first. A test sends order:1001 to an endpoint named acct:7 to prove it. The encoding covers the hyphen as well, because the hyphen that joins the two parts must be the only one. With a bare hyphen, event order-1 to endpoint customer and event order to endpoint 1-customer would share an id, and the second event’s transaction would fail. INSERT OR IGNORE on the event id makes a retried emit harmless as well.

Give each endpoint its own queue

BullMQ’s global concurrency and rate limit apply per queue, across every worker process. Per-customer limits inside one queue are groups, a feature of BullMQ Pro, the commercial edition. A queue per endpoint gets the same isolation from the open source package.

  async function queueFor(endpointId: string): Promise<AnyQueue> {
    const name = queueName(lane(endpointId))
    let queue = queues.get(name)
    if (!queue) {
      queue = backend.queue(name)
      queues.set(name, queue)
      // The cap holds across every worker process, not only this one.
      await queue.setGlobalConcurrency(concurrency)
      workers.push(backend.worker(name, (job) => deliver(job), {
        concurrency,
        settings: { backoffStrategy: (attemptsMade: number) => retryDelay(attemptsMade, scheduleMs, random) },
      }))
    }
    return queue
  }

  async function enqueue(deliveryId: string, endpointId: string): Promise<void> {
    // The delivery id is the job id, so enqueueing the same delivery twice adds one job.
    await (await queueFor(endpointId)).add('deliver', { deliveryId }, { ...jobOptions, jobId: deliveryId })
  }

  // Called from the request path: two inserts and a queue write, no HTTP.
  async function emit(event: { id: string; type: string; data: unknown }): Promise<string[]> {
    const payload = { type: event.type, timestamp: new Date().toISOString(), data: event.data }
    const ids = store.createEvent(event.id, event.type, payload)
    for (const id of ids) await enqueue(id, store.delivery(id)!.endpoint_id)
    return ids
  }

  // A crash between the insert and the queue write leaves a delivery with no job.
  // Run this on a timer to put such deliveries back on their queue.
  async function sweep(olderThanMs = 60_000, now = Date.now()): Promise<number> {
    const stranded = store.pendingSince(now - olderThanMs)
    for (const d of stranded) await enqueue(d.id, d.endpoint_id)
    return stranded.length
  }

setGlobalConcurrency caps a queue across all workers, so two endpoint slots stay two however many processes you run. The retry waits come from a custom backoff strategy that follows the specification’s example schedule, nine waits from five seconds to a day. The specification asks for random jitter without naming an amount, and the sample adds 20% either way.¹

On Redis, BullMQ keeps that cap and every job under a key prefix, so the cap holds only across processes that share the prefix. backend.ts reads it from BULLMQ_PREFIX and falls back to bull, BullMQ’s own default. The demo and the tests pass isolate: true for a prefix of their own, deleted on close, so a second run never meets the first run’s job ids.

Send each attempt with a timeout and log it

Each job sends one signed POST, writes one attempt row, and lets BullMQ schedule the retry.

  async function deliver(job: Job<{ deliveryId: string }>): Promise<string> {
    const d = store.delivery(job.data.deliveryId)
    if (!d || d.status !== 'pending') return 'skipped'
    if (d.endpoint_status !== 'enabled') {
      store.finish(d.id, 'cancelled')
      return 'endpoint disabled'
    }

    const n = job.attemptsMade + 1
    const at = new Date()
    const started = performance.now()
    let status: number | null = null
    let error: string | null = null
    let response = ''
    try {
      const res = await fetch(d.url, {
        method: 'POST',
        headers: {
          'content-type': 'application/json',
          'webhook-id': d.event_id, // the same on every attempt, so the receiver can deduplicate
          'webhook-timestamp': String(Math.floor(at.getTime() / 1000)),
          'webhook-signature': new Webhook(d.secret).sign(d.event_id, at, d.payload),
        },
        body: d.payload,
        redirect: 'manual', // a 3xx is a failed attempt, never a request to a host you did not check
        signal: AbortSignal.timeout(timeoutMs),
      })
      status = res.status
      response = await readCapped(res, 200)
    } catch (err) {
      if (status === null) {
        error = (err as Error).name === 'TimeoutError'
          ? `no response within ${timeoutMs} ms`
          : `connection failed: ${(err as { cause?: { code?: string } }).cause?.code ?? (err as Error).message}`
      }
    }
    store.recordAttempt({
      delivery_id: d.id, n, at: at.toISOString(), status, error,
      duration_ms: Math.round(performance.now() - started), response,
    })

    if (status !== null && status >= 200 && status < 300) {
      store.finish(d.id, 'delivered')
      return 'delivered'
    }
    if (status === 410) {
      // The receiver has asked to stop. Disable the endpoint rather than retry.
      store.disableEndpoint(d.endpoint_id)
      store.finish(d.id, 'cancelled')
      throw new UnrecoverableError('410 Gone: endpoint disabled')
    }
    if (n >= (job.opts.attempts ?? 1)) store.finish(d.id, 'failed')
    throw new Error(error ?? `HTTP ${status}`)
  }

Three choices in that function follow the specification’s advice on status codes. A 2xx is success and anything else is a failure. A 3xx is not followed, because a redirect sends your signed payload to a host nobody checked. A 410 disables the endpoint instead of retrying it.² The fourth is the 200-byte cap on the logged response, because an endpoint can answer 200 and then stream for as long as you let it.

The ten-second timeout matches what GitHub gives receivers. The specification recommends 15 to 30 seconds, and Svix allows 30. A shorter timeout frees worker slots sooner, and a receiver that follows the advice to acknowledge first needs milliseconds.

Run one event against four endpoints

The demo emits invoice.paid to four endpoints: one healthy, one that fails twice before succeeding, one that never answers, and one that answers 410. The schedule is scaled to waits of 50, 100 and 200 ms, and the timeout to 300 ms.

node --no-warnings demo.ts

queue backend: postgres (PGlite)

endpoint  try  result                       response
ep_a      1    200                          {"received":true}
ep_b      1    503                          upstream busy
ep_b      2    503                          upstream busy
ep_b      3    200                          {"received":true}
ep_c      1    no response within 300 ms
ep_c      2    no response within 300 ms
ep_c      3    no response within 300 ms
ep_c      4    no response within 300 ms
ep_d      1    410                          this endpoint was removed

deliveries: ep_a delivered, ep_b delivered, ep_c failed, ep_d cancelled
endpoints disabled: ep_d
requests received: 9, all verified: true, distinct webhook-id values: 1

The table is the attempt log a customer asks for when their integration missed an event. Each row also stores the attempt’s start time and duration, which vary from run to run and are left out of the printout. Every request carried one webhook-id and verified against the receiver’s secret with the standardwebhooks library. ep_c used all four attempts and failed; ep_d was switched off after one.

The same file run with REDIS_URL against Redis 7.0.15, in the session that wrote this page and again in review, printed the same lines apart from the backend label.

Check it worked

The suite pins the cases a customer would report: the signature and its webhook-id, a timeout, a 410, a redirect, and an endless body. It also covers a duplicate event, an id holding a colon, an id holding a hyphen, the fields backend.ts reads from REDIS_URL, the shared key prefix, and the sweeper.

test('the sweeper enqueues a delivery whose queue write never happened', async () => {
  const t = await setup(['ep_a'])
  // As if the process died between the insert and the enqueue.
  const [id] = t.store.createEvent('evt_t6', 'invoice.paid', { type: 'invoice.paid', data: {} }, Date.now() - 120_000)
  assert.equal(await t.sender.sweep(60_000), 1)
  await until(() => t.status(id) === 'delivered')
  assert.equal(await t.sender.sweep(60_000), 0)
  await t.done()
})

node --test sender.test.ts

# tests 12
# suites 0
# pass 12
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 5052.468398

The sweeper test fakes the crash the queue design leaves open. The rows commit and the process dies before the queue write, so without the sweeper the delivery would stay pending for good.

When it goes wrong

The API’s response time tracks the slowest customer, or one customer’s outage delays everyone. Sending inline causes the first, and one shared queue causes the second. pitfall.ts shows both.

node --no-warnings pitfall.ts

1. POST /orders with three subscribed endpoints, one of them hanging (1 s timeout)

inline   answered in 1 s or more   failed deliveries, never retried: 2
queued   answered in under 1 s     after the retries: ep_a delivered, ep_b delivered, ep_c failed

2. Three events to ep_a, ep_b and ep_c, where ep_c hangs (1 s timeout), concurrency 2

one queue for all endpoints  ep_a and ep_b deliveries finished before ep_c's first timeout: 4 of 6
one queue per endpoint       ep_a and ep_b deliveries finished before ep_c's first timeout: 6 of 6

The inline handler waited out the hanging endpoint and dropped two deliveries for good. The queued handler answered at once and retried in the background. With one shared queue, two hanging deliveries to ep_c held both worker slots, so two of the six healthy deliveries waited a full timeout.

With thousands of endpoints, the workers hold thousands of Redis connections. Each BullMQ Worker opens a blocking connection of its own, so a worker per endpoint per process does not scale without limit. Group endpoints into a fixed number of lanes, or move to BullMQ Pro’s groups.

With BullMQ’s default worker settings, retries on PGlite arrived about five seconds late, whatever the schedule said. An idle worker there noticed a delayed job only when its drainDelay wait of five seconds ran out, so backend.ts shortens it to 50 ms for the demos. In a run during review, four of the five retries to ep_b and ep_c came five seconds late with the default. Redis needs no such change: the same run there kept to the schedule.

When not to do this

Do not run this to send a few hundred webhooks a day to a handful of customers. Svix’s free tier covers 50,000 messages a month, with a portal your customers can use to replay what they missed.

Do not let a worker POST to any URL a customer types. The specification calls webhook senders especially exposed to server-side request forgery, and recommends sending through a proxy that refuses internal addresses.

Do not drop the sweeper because the queue write rarely fails. It fails when a deploy kills the process between the commit and add, and without the sweeper that delivery is never sent.

Do not add BullMQ to send webhooks if Postgres is your only data store. pg-boss, or BullMQ’s own PostgreSQL backend, avoids a second system. BullMQ’s PostgreSQL guide puts that backend at about 1.5 to 2 times fewer jobs a second than Redis on similar setups, depending on the workload, which is the cost to weigh.

Related how-tos

• Limit concurrency in a message worker

• Verify a Standard Webhooks signature in PHP

• Make queue consumer writes safe under redelivery

• Acknowledge a webhook fast and process it from a queue

Last verified

Verified 2026-10-15 against Node 22.22.0 and 22.23.3, BullMQ 6.3.9, ioredis 5.11.1, standardwebhooks 1.1.1, pg 8.23.1, PGlite 0.5.10, and pglite-socket 0.2.13. Every output block is what the command before it printed, on both Node versions, with BullMQ on its PostgreSQL backend over PGlite. The Redis run is described, not captured.