How-to › Operate integrations in production

How to kill-switch one vendor integration without a deploy#

Stop every call to one vendor within seconds, return a typed refusal instead of a slow failure, and decide whether a refused call queues, degrades, or errors.

Audience
API consumer
Level
intermediate
Topic
Control outbound integrations from one place
Languages
TypeScript
Verified

A vendor starts corrupting the records you send it at three in the morning, and stopping the calls takes a forty-minute deploy. The on-call engineer disables the vendor’s route at the gateway instead. The gateway answers 503, the SDK retries every 503 with backoff, and each checkout waits seven seconds for its refusal.

What you get

You will end up with a switch per vendor that refuses calls before they leave, one typed error for every kind of switch, and a policy per operation for refused calls. This is for you if containing a vendor incident has ever needed a deploy.

Short answer

Put the switch where your code can turn its refusal into one typed error. That can be an OpenFeature flag checked before the SDK call, a station policy in block mode, or a gateway route that answers with a body naming the switch. Keep every retry layer from retrying that error, or the switch becomes a slow failure. Decide per operation whether a refused call queues, degrades, or errors.

You will need

Node 22 or later, an SDK generated by sdkgen, and a staging copy of the vendor route to flip. Verified 2026-10-15 against Node 22.22.0, @voxgig/station 0.1.0, @voxgig/sdkgen 4.34.7 and @openfeature/server-sdk 1.23.0. The SDK under sdk/ was generated once by generate.sh with the retry feature and is checked in as compiled JavaScript. Unleash, LaunchDarkly and Kong did not run: the flag provider is OpenFeature’s in-memory provider, and the gateway is a loopback stand-in that terminates a route the way Kong’s plugin does. This page follows registering each SDK under one control surface, the previous step in the series. @voxgig/sdkgen 4.34.7 declares Node 24 or later in its engines field: generate.sh ran under Node 22.22.0, where npm warned and the build passed.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Disabling the gateway routeEvery caller reaches the vendor through one gateway, including services you cannot rebuildKong’s default 503 gets retried, so the refusal needs another status and a body clients recognizeCallers bypass the gateway, or nobody on call holds Admin API access
LaunchDarklyYou already pay for it and want its kill switch flag type for third-party toolsA paid service on the decision path, and a check your code makes before every callThe switch must also stop processes whose code you cannot change
OpenFeature with UnleashYou self-host flags and want a vendor-neutral API in your codeFifteen seconds of polling delay by default, and a flag service to keep runningThe switch must also stop processes whose code you cannot change
station block modeCalls leave through sdkgen SDKs, and you want the refusal before any connection opensNo deny rule, only a block mode read once at construction, so a flip needs a restart; 0.1.0 also refuses a live client with the SDK’s retry feature onA flip must take effect without a restart, or calls bypass the SDKs

Voxgig maintains station. station 0.1.0 has no deny rule: its policy holds allowlists of hosts and operations, plus a mode, and section 16 of its design document names mode: block as the kill switch. This page compares that block mode with an OpenFeature flag served by Unleash or LaunchDarkly, and with disabling the route at the egress gateway.

Speed and authority pull in opposite directions. A flag reaches a running process on the provider’s next poll or stream message, and anyone with a dashboard role can flip it.1 A gateway change reaches every caller, including code you do not own, but only someone with Admin API access can make it. station refuses earliest, before a socket opens. Its policy lives in a reviewed file, so a flip waits for a merge and a restart, or at best a restart under a profile written in advance.

Turn every switch into one typed refusal

Each switch refuses differently, so translate all three into one error before anything decides what to do next:

/**
 * Recognize a switch's refusal and return it as a VendorDisabled, or null for any other error.
 * station 0.1.0 refuses with station_host_allow, the code it also uses for a host missing from
 * the allowlist, so only a message naming block mode counts: a moved base URL is not a switch.
 * The gateway refuses with a body that names the switch, because its status alone (503 by
 * default for Kong's Request Termination plugin) means "try again".
 */
export function refusal(err, vendor) {
  if (err instanceof VendorDisabled) return err
  if (err?.code === 'station_host_allow' && /policy mode "block"/.test(err.message)) {
    return new VendorDisabled(vendor, 'station', err)
  }
  if (err?.result?.body?.kill_switch === vendor) return new VendorDisabled(vendor, 'gateway', err)
  return null
}

/**
 * The flag check, made before the SDK is touched. The default is the answer when the flag
 * service cannot be reached: true keeps the vendor on through a flag outage.
 */
export async function gate(flags, vendor) {
  const enabled = await flags.getBooleanValue(`${vendor}.enabled`, true)
  if (!enabled) throw new VendorDisabled(vendor, 'flag')
}

station’s block mode reuses station_host_allow, the code it raises for a host outside the allowlist, so refusal also checks that the message names block mode. The gateway’s status cannot carry that meaning, because Kong’s Request Termination plugin defaults to 503, which RFC 9110 defines as a condition that will likely be alleviated after some delay.2

Flip each switch in staging

The demo flips each switch and makes one call. Backoff is the sleep each retry policy asked for, recorded through an injected sleep rather than waited out:

node demo.mjs
the switch on, one Meter().list() per row         tries  backoff   the caller got
  station block mode                              1      0 ms      VendorDisabled via station
  station, app retry on every error               4      3500 ms   VendorDisabled via station
  station, app retry that skips refusals          1      0 ms      VendorDisabled via station
  gateway route, Kong's default 503, SDK retry    4      7000 ms   request_status 503
  gateway route, 403 naming the switch, SDK retry 1      0 ms      VendorDisabled via gateway
  OpenFeature flag off, checked first             0      0 ms      VendorDisabled via flag
  requests that reached the vendor in all six rows: 0
  station with the SDK's own retry feature on: station_wrap_order at construction

Every switch stopped the traffic, but two of them took seconds to say so. A retry wrapper that retries every error spent three and a half seconds being refused by a policy that will say the same thing every time. The SDK’s retry feature, set the way a production client might be, spent seven seconds on a bare 503 and then reported it as an outage. Name the switch in the body, use a status the SDK does not retry, and the refusal arrives in one try.

The last line is a version constraint. station 0.1.0 requires its wrapper to sit directly on the SDK’s base transport. An SDK from sdkgen 4.34.7 has no station feature of its own, so station appends its wrapper after the SDK’s features. A live client with the retry feature on then fails to bind. With station, retries belong in your code, behind a retryable test that excludes refusals.

Measure how a flip reaches a running process

The same demo flips each switch under a process that is already running:

node demo.mjs
flipped while the process runs, then the next call
  OpenFeature flag set to off                     VendorDisabled via flag
  gateway route terminated                        VendorDisabled via gateway
  station.json set to block, same Station         sent to the vendor
  station.json set to block, new Station          VendorDisabled via station

station reads station.json once, when the Station is constructed. Editing the file changes nothing until the process restarts. A profile written in advance and selected with VOXGIG_STATION_PROFILE, as section 3.5 of the design document describes, is the fastest station flip, and it still costs a restart. The in-memory flag provider here is immediate. The Unleash Node SDK polls every 15 seconds by default, and a Kong cluster in traditional mode spreads an Admin API change on a five-second poll.3

Decide what a killed call does

A refused call needs an answer chosen before the incident: queue it, degrade it, or fail it. The policy handles refusals only and throws any other error on untouched, so a vendor outage is never queued as if it were a switch:

export async function guarded(op, fn, { vendor, policy, outbox, cache, payload, key = op }) {
  try {
    const value = await fn()
    cache?.set(key, value)
    return { status: 'sent', value }
  } catch (err) {
    const off = refusal(err, vendor)
    if (!off) throw err
    if (policy === 'queue' && outbox) {
      // Replay later with an idempotency key, or a queued write can land twice.
      outbox.push({ op, payload, via: off.via })
      return { status: 'queued' }
    }
    if (policy === 'degrade' && cache?.has(key)) return { status: 'degraded', value: cache.get(key) }
    throw off
  }
}
node demo.mjs
the flag off, each operation under the policy decided for it
  meter.list     degrade  degraded, 1 meter from the cache
  meter.create   queue    queued, 1 write in the outbox
  meter.load     error    VendorDisabled via flag

Check it worked

The suite flips station in staging and asserts the caller gets a typed refusal in under a second with nothing sent. It counts the tries each retry policy makes, pins the restart rule, and checks that an allowlist refusal is not queued as a switch. The retry test is the one that catches the slow switch:

  for (const [retryable, tries, backoff] of [[undefined, 4, 3500], [(e) => !refusal(e, 'meterco'), 1, 0]]) {
    const waits = []
    let n = 0
    await assert.rejects(withRetry(() => { n++; return sdk.Meter().list() }, { retryable, sleep: async (ms) => { waits.push(ms) } }))
    assert.equal(n, tries)
    assert.equal(waits.reduce((a, b) => a + b, 0), backoff)
  }
node --test killswitch.test.mjs
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0

When it goes wrong

Calls fail with station_host_allow after a deploy, with no switch flipped. station 0.1.0 raises that code for a host missing from the allowlist too, so a moved base URL raises it. refusal takes it for the switch only when the message names block mode, so the error reaches the caller instead of the outbox. The match is on message text, so pin station and keep the test that covers it.

The vendor goes dark when the flag service does. The flag’s default is the answer during a flag outage, and a default of false turns it into a vendor outage. Default to on, as gate does.

Queued writes land twice after replay. The first attempt may have reached the vendor before the switch was flipped. Send each queued write with the idempotency key it was created with.

When not to do this

Do not switch off a vendor route with a bare 503. Every client that retries server errors will retry it, and the switch turns an instant refusal into seconds of backoff per call.

Do not choose station when the switch must work without a restart. Its policy is read at construction, so a flip means a merge to station.json and a restarted process.

Do not queue a write that cannot be replayed safely, or degrade a read that has nothing cached. Inventing an answer is worse than refusing.

Do not give the switch to someone who cannot see its effect. Whoever flips it at three in the morning needs the outbox size and the refusal count on the same screen.

Last verified

Verified 2026-10-15 against Node 22.22.0, @voxgig/station 0.1.0, @voxgig/create-sdkgen 0.30.7, @voxgig/sdkgen 4.34.7 and @openfeature/server-sdk 1.23.0. Every output block is what the command preceding it printed. The gateway and the vendor are loopback stand-ins, and the flag provider is OpenFeature’s in-memory provider.

Footnotes

  1. LaunchDarkly’s kill switch documentation describes them as permanent boolean flags, also called circuit breaker flags, for turning off third-party tools in an emergency. They serve “Enabled” while targeting is on and “Disabled” when it is off, so the emergency action is to switch targeting off. A flag built to be turned off stays in the codebase for good, waiting. ↩︎ Back to text

  2. The full sentence in RFC 9110 blames a temporary overload or scheduled maintenance and invites a Retry-After. A switched-off vendor is neither, which is why a client written to the specification does the reasonable thing and comes back. Kong’s plugin chose the status the specification reserves for coming back later. ↩︎ Back to text

  3. Kong’s configuration reference sets db_update_frequency to five seconds. When one node changes an entity through the Admin API, the others wait for their next poll to purge the cached copy. A route switched off on one node stays on for the rest until they ask. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.