The quarterly review shows sign-ups up 40% but cannot say whether the API got easier to use. The dashboard counts page views, sign-ups, and total requests, and all three moved because a conference talk went well. Meanwhile a third of new keys never make a successful call, which the dashboard does not show.
What you get
You will end up with a short scorecard whose numbers move when you change the product, plus a ranking of what your support queue is actually spending time on. This is for you if you own developer experience and need to report on it.
Short answer
Measure time to first successful call, the share of developers who ever succeed, the share who never call at all, and the error rate on early calls. Add two counts from your own surface: operations carrying an example, and operations documenting their failures. Then read where support time goes, because that is the list of things developers could not work out alone.
You will need
Node 22 or later, and access to sign-up and API request logs. Everything here is computed from events you already emit. The definitions borrow from the way the DORA program treats delivery metrics: few numbers, each tied to an outcome, each able to get worse when another improves.1 The Space framework makes the same argument about developer productivity, and warns against reporting any single number alone.2
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| A developer survey | You want to know why, and a number alone will not tell you | Response bias towards the people still using you, and a slow cadence | You need a signal that moves within a release |
| Funnel metrics from your own logs | Every API, because the events already exist and nobody has to be asked | Definitions you must agree on and keep stable, or the trend means nothing | The question is why developers left rather than how many |
| Session replay on the docs site | A specific page you suspect, watched for a week | Privacy obligations, and a view of the docs rather than of the API | The trouble is in the API rather than in the documentation |
| Support ticket analysis | You already have tickets, and the causes are cheap to label | Only the developers who asked, which is a minority of the ones who struggled | Nobody is labeling tickets, and nobody will |
Logs and tickets answer different questions and the pair is stronger than either. Logs say how many developers stalled. Tickets say what they stalled on. A team with only the first knows the number got worse and not what to fix, and a team with only the second fixes whatever the loudest customer mentioned.
Pick metrics that can disagree with each other. Time to first success can improve while the share who succeed falls, and that pattern is common. The developers who would have taken an hour gave up instead. The median got faster because they stopped appearing in it. A scorecard with one number hides that.
Count the people who never arrive
The developers who leave no trace are the most important row, and the hardest to see.
'share who ever succeed': ratio(succeeded.length, events.length),
'share who never call at all': ratio(events.length - called.length, events.length),A median computed over the developers who succeeded is a survivor statistic. It describes the people who got through, which is exactly the group whose experience needed the least improvement. Report it beside the two shares, always, or it will be read as the whole story.
Someone who signs up and never makes a request hit a problem before the API. A key they could not find, a quickstart assuming a tool they do not have, or a sign-up flow that ended somewhere other than a code sample. That number is often the largest one on the scorecard.
Measure your own surface, not only the traffic
Two counts you can compute today, with no telemetry at all.
'operations with an example': ratio(surface.operations_with_examples, surface.operations),
'operations documenting failures': ratio(surface.operations_with_error_documentation, surface.operations),These two predict outcomes rather than measuring them, and they earn their place because you control them directly. An operation with no example is a support ticket waiting to be filed, and the count tells you how many are queued.
Keep the list of languages developers asked for beside the list you ship. That ratio is the one that explains a stubborn share of sign-ups who never call, and it is invisible in every other number.
Check it worked
Compute the scorecard over a small cohort.
node demo.mjstime to first success (median seconds) 580
share who ever succeed 0.67
share who never call at all 0.17
first-hour error rate 0.21
operations with an example 0.62
operations documenting failures 0.36
languages shipped of those wanted 0.33
support tickets by cause
auth 4 0.4
missing-sdk 2 0.2
pagination 2 0.2
errors 1 0.1
rate-limits 1 0.1Read the first three lines together. A median of 580 seconds sounds tolerable until you notice that a third of the cohort never got there at all, and a sixth never made a request. Then read the ticket table: authentication is 40% of the support load, which is where the next week of work belongs, and it is nowhere in the traffic metrics.
The surface rows are the ones to act on first, because they need no telemetry pipeline and no debate about definitions. Two in five operations carry no example and two in three document no failure, and both of those are work somebody can start this afternoon.
node --test metrics.test.mjs1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 142.072414When it goes wrong
The numbers jump when nothing changed. A definition moved, or a load test signed up a hundred keys. Exclude your own traffic explicitly, and record the definition beside the number.
Time to first success improves every quarter and support does not get quieter. The metric is measuring the developers who were always going to succeed. Watch the share who succeed at the same time.
The error rate looks fine and developers are struggling. Errors per call hides a developer making four hundred calls. Report the share of developers whose first hour was mostly errors as well.
Nobody acts on the scorecard. It has fourteen rows. Cut it to five, put them on one screen, and attach an owner to each. A metric with no owner is a metric nobody defends when it gets inconvenient.
When not to do this
Do not measure what you cannot change this quarter. A number nobody can move becomes an item people explain away, and explaining it away becomes the ritual. Start with the numbers your next two releases could plausibly move.
Do not treat a survey score as the headline. It moves slowly, it is answered by the people who stayed, and it cannot tell you which of last month’s four changes helped.3
Do not instrument developer behavior beyond what you would be comfortable describing in your documentation. Trust is part of developer experience, and a team that finds a replay recorder on a docs page loses more than the metric was worth.
Related how-tos
Last verified
Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.
Footnotes
-
The four keys are five. The link on this page asks for the four keys, and the site redirects it to a page about five, which describes shifting from the original four keys to a five-metric model. The five are deployment frequency, change lead time, change fail rate, failed deployment recovery time and deployment rework rate. One of them was MTTR before it was renamed, and the page keeps a history of its own metrics for anyone who wants to know why. DORA is a program run by Google, and the footer of every page licenses the content under CC BY 4.0. ↩︎ Back to text
-
Space is an acronym, and the paper spells it in capitals. The SPACE of Developer Productivity, subtitled There’s more to it than you think, is by Forsgren, Storey, Maddila, Zimmermann, Houck and Butler. It appeared in ACM Queue in February 2021, volume 19, pages 20 to 48. Its abstract states that developer productivity cannot be measured by a single metric or dimension, a sentence six authors agreed on. The scorecard on this page has seven rows, which is one way of taking the advice. ↩︎ Back to text
-
Stack Overflow’s survey prints its own denominator. The 2025 edition reports more than 49,000 responses from 177 countries, across 62 questions about 314 technologies. One of its headline findings is that one in four developers are happy at their current job, with 28.4% not happy and the remainder described as complacent. Every one of those numbers describes the people who chose to answer, which is the observation this page makes about a survey score, at a scale of 49,000. ↩︎ Back to text