How to choose the layer that instruments outbound API calls#

Your panel of outbound calls to one vendor doubles overnight while traffic stays flat. A team switched on its SDK’s telemetry in a service the platform’s OpenTelemetry setup already instruments. Each call now produces two spans in two traces, three when it is retried, and the proxy log cannot say which tenant made any of them.

What you get

You will end up with one retried call captured at four layers side by side, a rule for which layer owns the span, and tests that hold the rule. This is for you if other teams instrument their services against the platform you run.

Short answer

Make one call that fails and is retried, and capture it with OpenTelemetry’s HTTP instrumentation, the SDK’s own telemetry, the egress proxy’s access log, and the gateway’s log. Only an in-process layer can name the operation and the tenant. Give the span one in-process owner, number attempts with http.request.resend_count, and keep the proxy and gateway logs for the processes you cannot rebuild.

You will need

Node 22 or later, an SDK generated by sdkgen, and somewhere to send spans. Verified 2026-10-15 against Node 22.22.0, @voxgig/create-sdkgen 0.30.7, @voxgig/sdkgen 4.34.7, and the OpenTelemetry JS packages listed in the page’s versions. The SDK under sdk/ was generated once by generate.sh with the telemetry, metrics, and retry features, and is checked in as compiled JavaScript. @voxgig/sdkgen 4.34.7 declares Node 24 or later in its engines field, and generate.sh ran under Node 22.22.0, where npm warned and the build passed. Envoy and Kong did not run: the proxy and the gateway are Node servers on 127.0.0.1 that write a subset of the fields each product logs by default. If you have not instrumented an HTTP client with OpenTelemetry before, do that first.

Approaches compared

Voxgig maintains sdkgen. This page compares its telemetry and metrics features with OpenTelemetry auto-instrumentation, an Envoy egress proxy, and Kong’s gateway logs. In sdkgen 4.34.7 the telemetry feature model describes W3C trace-context propagation, but its default ids fail the W3C format. The metrics feature takes no exporter, and the generated README documents neither feature’s options beyond active.

The trade-off is fidelity against coverage. OpenTelemetry and the SDK features run inside the process, so they know the operation and the tenant, but only for processes you rebuilt. Envoy and Kong see every process on the route, including an agent you cannot change, but only as bytes. A gateway also costs a hop per call, and a proxy cannot carry its own request id forward.¹

Capture one retried call at every layer

The call path is SDK, egress proxy, gateway, vendor, and the vendor answers 503 once. Node stand-ins from stands.mjs play the proxy and the gateway, since Envoy and Kong did not run. The SDK’s features are switched on in its constructor:

export function meterco({ base, telemetry = true, idgen } = {}) {
  const sdkSpans = []
  const client = new MetercoSDK({
    apikey: 'sk_test_meterco',
    base,
    feature: {
      retry: { active: true, jitter: false },
      metrics: { active: true },
      telemetry: {
        active: telemetry,
        ...(idgen ? { idgen } : {}),
        exporter: (span) => sdkSpans.push({ ...span, tenant: callContext.getStore()?.tenant }),
      },
    },
  })
  return { client, sdkSpans }
}

The tenant comes from an AsyncLocalStorage that your code sets. A span processor copies it onto the OpenTelemetry spans the same way. Both are code you write, and both are only possible because the layer shares the caller’s process.

node capture.mjs

one call: Meter().list() for tenant acme, vendor answers 503 then 200

OpenTelemetry undici, 2 records
  {"name":"GET","status":503,"error":true,"path":"/meters","tenant":"acme"}
  {"name":"GET","status":200,"error":false,"path":"/meters","tenant":"acme"}
sdkgen features, 3 records
  {"span":"meter.list","ok":true,"tenant":"acme"}
  {"metrics":"meter.list","count":1,"ok":1,"err":0}
  {"retry":true,"attempt":1,"status":503,"wait":50}
egress proxy (Envoy fields), 2 records
  "GET /meters HTTP/1.1" 503 - "Mozilla/5.0 (compatible; MetercoSDK/1.0)" "f403dc9c-9363-43db-a2d5-d722aaefcfc5" "127.0.0.1:46645"
  "GET /meters HTTP/1.1" 200 - "Mozilla/5.0 (compatible; MetercoSDK/1.0)" "2b45d93c-3d09-4f63-ace7-838644c13d6d" "127.0.0.1:46645"
gateway (Kong http-log fields), 2 records
  {"request":{"method":"GET","uri":"/meters"},"response":{"status":503},"service":{"name":"meterco"},"tries":1,"consumer":null}
  {"request":{"method":"GET","uri":"/meters"},"response":{"status":200},"service":{"name":"meterco"},"tries":1,"consumer":null}

can the layer name                operation  tenant  attempt
  OpenTelemetry undici            no         yes     no
  sdkgen features                 yes        yes     yes
  egress proxy (Envoy fields)     no         no      no
  gateway (Kong http-log fields)  no         no      no

The matrix is computed from the records, not written by hand. The SDK names the operation because it knows it called list on Meter. Its attempt number comes from the retry feature’s record on client._retry, as undocumented as _metrics. The span is one per operation and reports the call as successful, with no trace of the 503. OpenTelemetry sees two unrelated GET requests. The proxy and the gateway see two requests with two request ids. Kong’s tries counts the gateway’s own balancer attempts, so it reads 1 on both.²

Give the span one owner

Switch both in-process layers on and the call splits. The pitfall script runs the same retried call four ways and reports what reached the vendor:

node pitfall.mjs

who writes traceparent                  spans  traces  headers   vendor finds a parent
  sdkgen telemetry and OTel undici      3      3       2, 2      no
  sdkgen telemetry, default ids         1      1       1, 1      no
  sdkgen telemetry, idgen with hex ids  1      1       1, 1      yes
  OTel undici under an operation span   3      1       1, 1      yes

traceparent as the vendor's server joined it, both layers on:
  00-t0001000000000000-s0002000000000000-01, 00-<32 hex>-<16 hex>-01
traceparent from sdkgen's default ids:
  00-t0001000000000000-s0002000000000000-01

With both layers on, each request carries two traceparent headers, so a panel counting client spans counts this one call three times. The vendor’s W3C parser finds no parent, because the joined value is not one header. The SDK’s default ids fail on their own as well. W3C Trace Context requires a trace id of 32 lowercase hex characters, so a parser ignores t0001000000000000, which has 16 characters and a t.³ The span id s0002000000000000 fails the parent id rule on its s. Pass an idgen that returns hex.

The fourth row is the rule written as code. Your own span owns the operation, and every HTTP attempt is a child of it:

export function operation(name, tenantId, fn) {
  const call = { tenant: tenantId, operation: name, sends: new Map() }
  return callContext.run(call, () =>
    tracer.startActiveSpan(name, { attributes: { 'tenant.id': tenantId } }, async (span) => {
      try {
        return await fn()
      } catch (err) {
        span.recordException(err)
        span.setStatus({ code: SpanStatusCode.ERROR })
        throw err
      } finally {
        let requests = 0
        for (const n of call.sends.values()) requests += n
        span.setAttribute('app.http_requests', requests)
        span.end()
      }
    }))
}

A request hook on the undici instrumentation numbers each repeated send per method and URL, as http.request.resend_count, which the semantic conventions ask for on every repeated span. The first send carries no attribute.

Check it worked

The suite proves the wire layers cannot name the operation or the tenant, and that two owners give two headers and three traces. It checks that a call with no retry still yields the SDK’s records, since client._retry does not exist until the first retry. It also checks that one owner gives one trace with numbered attempts:

  assert.equal(op.attributes['tenant.id'], 'acme')
  assert.equal(op.attributes['app.http_requests'], 2)
  assert.deepEqual(attempts.map((s) => s.attributes['http.response.status_code']), [503, 200])
  assert.deepEqual(attempts.map((s) => s.attributes['http.request.resend_count']), [undefined, 1])
  assert.equal(new Set(spans.map((s) => s.spanContext().traceId)).size, 1, 'one call, one trace')
  assert.deepEqual(api.state.requests.map(traceparents), [1, 1])

node --test layers.test.mjs

# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0

When it goes wrong

Span counts double after a team enables SDK telemetry. Two layers write traceparent. Pick one: leave feature.telemetry off, or switch the undici instrumentation off for that process.

A different request in the same operation shows up as a retry. The counter keys on the operation alone. Key it on method and URL, as the request hook here does, and a second test pins it.

The metrics never reach your backend. In sdkgen 4.34.7 the metrics feature has no exporter option: it keeps counters on client._metrics, which the generated README does not document. Read them on a timer and export them yourself, and pin the SDK version, since an underscore property can move.

When not to do this

Do not switch on the sdkgen telemetry feature in a process that already runs the OpenTelemetry undici instrumentation. Each attempt then sends two traceparent headers, and a call retried once starts three traces, so the vendor’s side cannot attach to any of them.

Do not send sdkgen’s default trace ids across a process boundary. Give the feature an idgen that returns hex, or leave the span to OpenTelemetry.

Do not attribute cost per tenant from the gateway unless each tenant authenticates as its own Kong consumer. With one shared credential, the gateway knows the service but not the caller.

Do not drop the proxy log because the in-process spans are richer. It is the only record of calls from processes you cannot rebuild, and the place to count requests the SDK never saw.

Related how-tos

• Export outbound call spans from a serverless function

• Tag SDK calls with session and request ids

• Route outbound API calls through an egress proxy

Last verified

Verified 2026-10-15 against Node 22.22.0, @voxgig/create-sdkgen 0.30.7, @voxgig/sdkgen 4.34.7, @opentelemetry/sdk-trace-node 2.11.0 and @opentelemetry/instrumentation-undici 0.32.0. Every output block is what the command preceding it printed. The proxy and the gateway are loopback stand-ins that record a subset of Envoy’s and Kong’s default fields, in place of the products themselves.