Nobody was paged when your payment provider had a bad afternoon. The only alert on that integration watches your own service’s error rate, which stayed under its threshold while the provider failed one call in ten. Two weeks later a deploy sent malformed requests, and the same alert paged the on-call for a provider outage that did not exist.
What you get
You will end up with an objective for one dependency, measured from your side of the wire. Its burn-rate alerts page for a real degradation and stay quiet for your own bugs. This is for you if you operate integrations you cannot fix yourself.
Short answer
Measure the dependency from your own client. Count calls the provider answered with a 5xx or not at all, over every call except the ones your code sent malformed. Those come back 4xx and belong to you, so leave them out of both sides. Set the objective over 30 days, alert on burn rate with a long and a short window, and write down what the team does when the budget is gone.
You will need
Node 22 or later to run the generator and the simulation, a client metric that counts calls by outcome, and Prometheus or Datadog to load the rules into. Verified 2026-09-25 against Node 22.22.2. The windows and burn rates come from the Google SRE workbook.1
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Datadog SLOs | Your metrics already live in Datadog and you want the burn-rate monitor in the UI | A long window in whole hours, a short window fixed at one twelfth unless set through the API, and a 7, 30, or 90 day target | Your metrics are in Prometheus, or you need a window the UI does not offer |
| Hand written Prometheus rules | One or two dependencies, and a team that wants every threshold in view | Five recording rules and two alerts per objective, each a place for the multiplication to be wrong | More than a handful of objectives, or nobody willing to own the arithmetic |
| Pyrra | You run Kubernetes and want objectives as resources, with a UI over them | An operator to run, a window written as a Prometheus duration rather than in days, and its own choice of alert windows | You are not on Kubernetes, where the file mode buys little over Sloth |
| Sloth | You want the workbook’s windows generated from a short spec, with no cluster | A generate step in CI and a second artifact to keep in step with the spec, with windows changed only through plugins | You want to read the rules you deploy without a generator between you and them |
A generator gets the multiplication right once and then asks you to keep two files in step. Sloth turns a twenty line spec into rules whose thresholds it computed, at the cost of a build step between you and the file Prometheus loads. Datadog computes the same thing inside its UI, at the cost of a window vocabulary you did not choose.
Pick an indicator the provider is responsible for
An objective for a dependency is a claim about what the provider did, so the indicator has to leave out what you did.
export const slo = {
name: 'billco',
objective: 0.995,
period: '30d',
metric: 'billco_client_requests_total',
// The provider failed: a 5xx, or no answer at all.
errors: 'code=~"5..|timeout|connect"',
// Every call except the ones the caller got wrong.
total: 'code!~"4.."',
}A timeout and a refused connection count as failures, because from your side a provider that never answered and one that answered 503 look the same. A 4xx is excluded from both the numerator and the denominator. In the denominator alone, your own bad requests improve the provider’s score. In the numerator, they blame the provider for your bug. A 429 could go either way, and the definition treats it as yours: watch it with an alert on your own quota use.
The objective is 99.5% over 30 days, a budget of one call in two hundred. Set it no higher than the provider commits to in writing, or you will miss it on their schedule.
Generate the rules and read the thresholds
The recording rules take the ratio over each window, named by the convention Prometheus documents for recording rules, and the alerts compare those ratios with the burn rate times the budget. Writing the generator is shorter than writing the rules.
const condition = (rows) => rows.map((w) => {
const t = threshold(slo, w.burnRate).toFixed(4)
return `(${ratioRule(slo, w.long)} > ${t} and ${ratioRule(slo, w.short)} > ${t})`
}).join('\n or\n ')Print the rules and read the numbers before loading them anywhere.
node rules.mjs - alert: BillcoErrorBudgetBurn
expr: |
(billco:slo_errors_per_request:ratio_rate1h > 0.0720 and billco:slo_errors_per_request:ratio_rate5m > 0.0720)
or
(billco:slo_errors_per_request:ratio_rate6h > 0.0300 and billco:slo_errors_per_request:ratio_rate30m > 0.0300)
labels:
severity: page
dependency: billco
annotations:
summary: billco is burning the 30d error budget at over 6x
- alert: BillcoErrorBudgetBurn
expr: |
(billco:slo_errors_per_request:ratio_rate3d > 0.0050 and billco:slo_errors_per_request:ratio_rate6h > 0.0050)
labels:
severity: ticket
dependency: billco
annotations:
summary: billco is burning the 30d error budget at over 1xThe page threshold of 0.072 is 14.4 times the 0.005 budget. An error ratio of 7.2% over the last hour, confirmed over the last five minutes, spends 2% of the month’s budget and wakes somebody. The ticket threshold is the budget itself. A provider failing one call in two hundred for three days is on course to spend the whole month, which is worth a ticket rather than a page.2
Load the file through
rule_files, and
keep a
promtool test rules
file beside it so a change to the expression fails in CI rather than on a bad afternoon.
Prove the alert blames the right party
Three days of synthetic traffic, evaluated two ways: a naive indicator that counts every non-2xx against the provider, and a corrected one that counts the codes the definition’s selectors match.
export const indicators = {
naive: (m) => ({ errors: m.e4xx + m.e5xx, total: m.ok + m.e4xx + m.e5xx }),
corrected: (m) => ({ errors: m.e5xx, total: m.ok + m.e5xx }),
}There is no Prometheus in the simulation: once rate() has been taken, a burn rate is arithmetic
on counts.
node simulate.mjsnode 22.22.2, three days of synthetic traffic at 200 calls a minute
objective 99.5% over 30d, so the budget is 0.5% of calls
page at 14.4x (error ratio 0.072) over 1h and 5m, or 6x over 6h and 30m; ticket at 1x over 3d and 6h
scenario indicator 1h/5m 6h/30m 3d/6h result
provider answers 503 for six minutes naive 20.0/200.0 3.3/40.0 0.3/3.3 page
corrected 20.0/200.0 3.3/40.0 0.3/3.3 page
your deploy sends 15% malformed requests naive 30.0/30.0 30.0/30.0 30.0/30.0 page
corrected 0.0/0.0 0.0/0.0 0.0/0.0 quiet
provider fails 0.6% of calls, all three days naive 1.2/1.2 1.2/1.2 1.2/1.2 ticket
corrected 1.2/1.2 1.2/1.2 1.2/1.2 ticketEach cell is the burn rate over the long and the short window of one alert pair. Six minutes of total outage is 200x over five minutes, the highest value a 99.5% objective can show, and both indicators page. The middle scenario is the one that matters. A deploy sending 15% malformed requests pages the naive indicator at 30x while the provider is healthy, and the corrected indicator reads zero. The slow burn at the bottom is a ticket under both.
Write the policy before the budget burns
An objective without a policy is a dashboard. The policy is a short document beside the rules naming what the team does at each level of budget spent. You cannot fix the provider. You can stop making things worse, and you can make the provider’s problem visible to the provider.
Three levels are enough. At half the budget gone with half the month left, open the conversation with the provider, with your indicator’s numbers rather than an impression. At a spent budget, stop shipping changes to the integration that are not reliability changes, and turn on whatever degrades gracefully: a circuit breaker, a cached answer, a retry later.
The workbook’s advice is to freeze changes even when the outage was somebody else’s, and to record the exception when freezing is impractical.3 For a dependency the exception is common.
Hand the same objective to a generator
Neither generator was run on this page: both are Go programs, and the specs below follow their
documentation. In Sloth’s spec, {{.window}} is where each generated window goes.
sli:
events:
error_query: sum(rate(billco_client_requests_total{code=~"5..|timeout|connect"}[{{.window}}]))
total_query: sum(rate(billco_client_requests_total{code!~"4.."}[{{.window}}]))
alerting:
name: BillcoErrorBudgetBurn
labels:
dependency: "billco"
annotations:
summary: "BillCo is burning the checkout error budget"
page_alert:
labels:
severity: page
ticket_alert:
labels:
severity: ticketsloth generate -i sloth.yml writes the rules. The output on its documentation site uses the
workbook’s page windows and adds a one day and two hour pair at 3x on the ticket side.
Pyrra takes a Kubernetes resource, or the same document from a file, with a Prometheus duration as the window.
spec:
target: "99.5"
window: 4w
description: Calls BillCo answered correctly, leaving out requests the caller sent malformed.
indicator:
ratio:
errors:
metric: billco_client_requests_total{code=~"5..|timeout|connect"}
total:
metric: billco_client_requests_total{code!~"4.."}Four weeks is 28 days, not 30. The recording rules Pyrra derives use windows of its own, which run from three minutes to two days for the two week example in its README. A generator’s window vocabulary is part of what you adopt.
Datadog’s metric-based objective is two queries, good events and bad events, so the exclusion goes into both, and its burn-rate monitor takes the long window in whole hours.
Check it worked
The suite pins the thresholds, the denominators, and the three scenarios. It also matches the selectors against real codes, so a definition that starts counting 4xx on either side fails here.
test('your own malformed requests page the naive indicator and not the corrected one', () => {
const s = scenarios['your deploy sends 15% malformed requests']
assert.equal(evaluate(s, indicators.naive).result, 'page')
assert.equal(evaluate(s, indicators.corrected).result, 'quiet')
})node --test slo.test.mjs1..10
# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 124.665466When it goes wrong
The page fires the morning after a deploy and the provider’s status page is green. A 4xx has found its way into the numerator, often through a client library that reports every non-2xx as an error. Split the metric by status and re-read the selector.
Nothing fires during a three minute total outage. The hour is at 5%, under the 7.2% threshold, so by design nobody is paged for 1.4% of the month’s budget. That case belongs to a circuit breaker, not the pager.
The alert flaps through a quiet night. With ten calls an hour, one failure is a 10% error rate and a 20x burn. The workbook covers the options for low-traffic services: a minimum request count in the expression, synthetic traffic, or a lower objective.
When not to do this
Do not set an objective for a dependency above what the provider commits to in its own terms. A 99.99% objective on a provider that promises 99.9% will be missed on a schedule you do not control, and every miss is a page with no action attached.
Do not count your own 4xx against the provider, whatever the client library reports as an error. The simulation shows what that costs: a page at 30x during a deploy, for an outage that was yours.
Do not page on a single window. One window either pages for a blip or misses a slow burn, and the second is what lets the alert clear five minutes after the provider recovers.
Do not skip the policy because the provider is outside your control. The actions are fewer and less obvious than for your own service, and deciding them during the outage is the expensive way.
Do not adopt a generator to save writing five recording rules. Adopt one when the arithmetic becomes a maintenance problem, and read the rules it produces, because its windows are now yours.
Related how-tos
Last verified
Verified 2026-09-25 against Node 22.22.2. Every output block is what the command preceding it printed. The rules were generated and the burn rates computed locally. No Prometheus, Sloth, Pyrra, or Datadog was run.
Footnotes
-
The 14.4 is not a tuning constant. Table 5-6 of the workbook derives it. The budget an alert spends is the burn rate times its window over the period. So 2% of a 30 day budget in one hour takes a rate of 0.02 times 720 hours: 14.4. The 6 and the 1 fall out of 5% in six hours and 10% in three days the same way. Change the period and every number moves. ↩︎ Back to text
-
A burn rate has a ceiling. Datadog’s documentation points out that an error rate cannot exceed 100%. The largest burn rate an objective can show is therefore one over one minus the target: 200 for 99.5%, 1,000 for 99.9%. It refuses to create an alert with a threshold past that, since such an alert asks to be told when more than every request has failed. ↩︎ Back to text
-
The chapter on implementing SLOs records two schools of thought on a missed objective caused by another team’s dependency. Carry on, since your system did not cause it, or freeze changes regardless. It sides with the freeze, on the grounds that the second approach will make your users happier. It then allows that freezing may not be practical, and asks only that the decision be recorded. A policy that says it depends is still a policy, once it says on what. ↩︎ Back to text