How to simulate slow and broken networks with Toxiproxy#

Your client sets a timeout, but in production a call to the pricing service still hangs for minutes. The headers arrived in time, then the body stopped halfway, after the timer had already been cleared. Your tests only ever ran against a healthy local dependency, so this path has never executed before the night it paged someone.

What you get

You will end up with a test suite that adds latency, throttles bandwidth, stalls a response and cuts a connection mid-body, and a client that fails at its budget in each case. This is for you if your service has timeouts nobody has watched fire.

Short answer

Put Toxiproxy between your service and the dependency, then add toxics from the test through its HTTP API on port 8474: latency, bandwidth, timeout and limit_data. Assert that a call past your time budget fails at the budget, and that a connection cut mid-response is an error rather than a short answer. Give the client one budget that covers the body as well as the headers.

You will need

Node 22 or later, a dependency reached over TCP, and Toxiproxy 2 wherever its server can run. Verified 2026-10-15 against Node 22.22.0 and Toxiproxy 2.13.0, built here from its Go module. The CI that re-runs this page cannot start toxiproxy-server, so the code directory carries an in-process stand-in that takes the same toxic objects. Set TOXIPROXY_URL and the same tests run against the real server instead; in this container both gave the same results.

Approaches compared

tc netem works below every protocol and asks for root, so a delay meant for one test reaches every process on that interface.¹ WireMock sits at the HTTP layer, so it can delay or garble a response but cannot stand in front of a database. Toxiproxy proxies bytes, so it works for Redis or Postgres as well, at the cost of a server and a changed connection address.²

Put Toxiproxy in the path

Start the server, create a proxy in front of the dependency, and point the client at the proxy’s listen address. The HTTP API takes toxics as JSON, and toxiproxy-cli wraps the same calls.

toxiproxy-server -host 127.0.0.1 -port 8474

toxiproxy-cli create -l 127.0.0.1:26000 -u 127.0.0.1:8080 pricing
curl -s -X POST http://127.0.0.1:8474/proxies/pricing/toxics \
  -d '{"type":"latency","attributes":{"latency":5000}}'

Run here, the curl answered with the stored toxic, named latency_downstream. Toxics are directional, and a latency toxic delays only the server-to-client stream unless you add one with "stream": "upstream" too.³

The tests reach either server through one function. With TOXIPROXY_URL set it creates a proxy through the API, and without it the stand-in starts.

export function openProxy(upstreamPort, name = 'pricing_test') {
  const api = process.env.TOXIPROXY_URL
  return api
    ? openToxiproxy(api, { name, upstreamPort })
    : startToxicProxy({ upstreamPort })
}

The stand-in implements four toxic types, latency with its jitter, bandwidth, timeout and limit_data, and refuses the others, slicer and reset_peer among them, with an error rather than a silent pass. Where the README is silent it follows the 2.13.0 source. Deleting a timeout toxic closes the connections it held, through the cleanup that toxic runs. Deleting a latency toxic releases what it was holding, bandwidth pays a large chunk out in 100 ms slices, and an upstream that answers and hangs up still gets its answer through a delay. Four tests hold the stand-in to those behaviors, and they passed against Toxiproxy 2.13.0 in this container too. It is a test fixture, not a replacement.

Give the client one budget for the whole response

The budget has to cover the body. A signal from AbortSignal.timeout passed to fetch keeps running while res.text() reads, so a stall after the headers still ends at the budget.

export async function getQuote(base, { budgetMs = 1000 } = {}) {
  const signal = AbortSignal.timeout(budgetMs)
  try {
    const res = await fetch(new URL('/v1/quote', base), { signal })
    const text = await res.text()
    if (!res.ok) throw new UpstreamError('status', `upstream answered ${res.status}`)
    return JSON.parse(text)
  } catch (err) {
    throw classify(err, budgetMs)
  }
}

classify turns Node’s errors into kinds a caller can act on. A connection closed before the headers rejects fetch with fetch failed. Closed after them, reading the body rejects with terminated. Both carry the socket error on err.cause, and both strings belong to Node’s fetch, so the tests pin them.

Add toxics from the test

Each test opens its own proxy, adds the toxic it needs, and asserts on the kind of failure and on how long it took. The margins are generous, because a timer that fires late on a loaded CI runner is not a defect. The mid-body stall waits on events instead of a guessed delay: the toxic goes in once fetch reports the headers, and only then does the upstream send the rest of the body.

test('latency past the budget fails at the budget, not at the latency', async () => {
  await proxy.addToxic({ type: 'latency', attributes: { latency: 5000 } })
  const { error, ms } = await timed(getQuote(proxy.url, { budgetMs: 400 }))
  assert.equal(error?.kind, 'timeout')
  assert.ok(ms < 3000, `failed after ${ms} ms, the budget is 400 and the latency 5000`)
})

test('a connection dropped mid-response is an error, not a short answer', async () => {
  await proxy.addToxic({ type: 'limit_data', attributes: { bytes: 300 } })
  const { error } = await timed(getQuote(proxy.url))
  assert.equal(error?.kind, 'dropped')
})

limit_data is the toxic most suites never add. It passes the first 300 bytes, the headers and part of the body, then closes the connection. A load balancer restart or an upstream crash produces that failure. It is the one that pages you, because latency alone never exercises the code that reads half a body.

Check it worked

Run the suite. Two tests are skipped against the real server, because they check the stand-in itself: that it refuses a toxic it does not implement, and that it applies jitter.

node --test slow-network.test.mjs | tail -9

1..14
# tests 14
# suites 0
# pass 14
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 6244.884074

Run the demo to see what the client reports for each toxic. The last row uses the timer most code reaches for, one cleared once the headers arrive.

node demo.mjs

backend: in-process stand-in
no toxics                              ok: quote q_1, 25 lines
latency 300 ms                         ok: quote q_1, 25 lines
latency 5000 ms, budget 400 ms         timeout: no complete answer within 400 ms, within 1 s of the budget: true
bandwidth 1 KB/s, budget 500 ms        timeout: no complete answer within 500 ms, within 1 s of the budget: true
limit_data 300 bytes                   dropped: connection closed mid-response: other side closed
timeout 0 mid-body, budget 600 ms      timeout: no complete answer within 600 ms, within 1 s of the budget: true
the same, timer cleared at headers     still waiting 2100 ms in, budget 600 ms

The bandwidth row throttles the response to 1 KB/s,⁴ so the 1,014-byte body cannot arrive inside 500 ms. With TOXIPROXY_URL pointing at Toxiproxy 2.13.0 in this container, the same command printed the same seven rows under backend: toxiproxy-server. Every timeout fails within a second of its budget, the cut connection is an error, and the cleared timer is still waiting.

When it goes wrong

The call hangs past its timeout. The timer was cleared when fetch resolved, which is when the headers arrived, so nothing bounds the body. Pass the signal to fetch and leave it in place until the body is read, as getQuote does.

A truncated response reaches your parser as if it were whole. A body framed only by the connection closing, with no Content-Length and no chunking, looks complete to fetch when it is cut. One test proves it: through limit_data, fetch resolved and only JSON.parse failed. A line-delimited format cut at a line boundary would parse cleanly. Send a length, or validate what you parse.

Tests pass alone but fail together. Run here against the real server, one shared proxy with its toxics deleted between tests failed two runs in six. A connection opened just after a timeout toxic was deleted closed before its headers. A proxy per test, created in beforeEach and deleted in afterEach, ran ten times clean.

WireMock covers part of this. Its MALFORMED_RESPONSE_CHUNK fault reached the same client as dropped, and CONNECTION_RESET_BY_PEER and EMPTY_RESPONSE failed before any response, in a run against WireMock 3.13.2 in this container. Its chunkedDribbleDelay produced the slow body. None of its faults sends valid headers and then half a valid body.

When not to do this

Do not put Toxiproxy in front of a dependency your test can fake in process. A stub that answers slowly or returns a broken body is faster to run and has no server to start. Toxiproxy earns its place when the real client library, its connection pool and the real protocol are what you are testing.

Do not leave the API reachable from other machines. The README states that it is unauthenticated by default and can create proxies to arbitrary upstreams, and the changelog for 2.13.0 adds TOXIPROXY_AUTH_TOKEN to require a token.

Do not use tc netem for a test suite on a shared machine. A delay on an interface applies to every process on it, including the CI agent’s own connection to its controller.

Do not trust the stand-in in this code directory beyond the four toxics it implements. Run the same suite against the real server wherever one can start.

Related how-tos

• Time out fetch calls with AbortSignal in Node.js

• Enforce one total deadline across all retry attempts

• Add a circuit breaker to an outbound HTTP client

• Control which stub wins when mock rules overlap

Last verified

Verified 2026-10-15 against Node 22.22.0. Every output block is what the command before it printed, against the in-process stand-in. Toxiproxy 2.13.0 and WireMock 3.13.2 ran in this container, and what they did is described in prose rather than shown as captured output.