How to audit a DevRel program in one week#

Nobody in the room can say what any of the twelve talks, twenty-four posts, and forty office hours in the quarterly review changed. The budget holder asks which activities to cut. The team answers with a longer list of activities. Every number on the slide counts effort, and effort is the one thing nobody was disputing.

What you get

You will end up with a scored report in which every activity line carries an outcome and a number, plus three changes for next quarter with the evidence behind each. This is for you if you run or fund a DevRel team and the next review is weeks away.

Short answer

Spend day one listing every activity with the purpose the team gives it. Spend days two to four replacing each line’s activity count with an outcome taken from evidence: response times, attributed sign-ups, sample app builds, event follow-ups. On day five score each line 0 to 3 against its target, then score the whole against the AAARRRP funnel and against Orbit gravity. Present the three lines furthest from target as next quarter’s changes.

You will need

Node 22 or later, one quarter of activity records, and read access to the sign-up data, the forum export and the sample app repositories. Verified 2026-09-24 against Node 22.22.2. The scoring borrows one rule from the DORA metrics: a number earns its place only if it can get worse when the team works harder. The five-day shape is older than DevRel.¹

Approaches compared

Speed and context sit on one side, independence on the other. The self-run audit is done by Friday and it knows why the forum went quiet in July, and it is written by the people being audited. The external audit is believed by the board and costs weeks, a fee, and a questionnaire that cannot know your product. The two scoring frames split the same way: the funnel counts what the business wanted, and gravity counts what the community did.

Day one: write down what the team does and why

The inventory is a file, not a slide. One line per activity, with the count the team reports, the count it planned, and the purpose it claims. The purpose is one of the seven AAARRRP stages: awareness, acquisition, activation, retention, referral, revenue, product.²

    {
      "activity": "Conference talks",
      "count": 12,
      "unit": "talks given",
      "planned": 12,
      "purpose": "awareness",
      "outcome": {
        "label": "sign-ups naming a talk in the referral field within 30 days",
        "value": 9,
        "target": 60,
        "source": "sign-up form, referral field"
      },
      "orbit": [{ "from": null, "to": 4, "members": 40, "reach": 2 }]
    },

On Monday the outcome field is empty on every line. Filling it is the rest of the week. The purpose is whatever the team says it is, written down verbatim, because an activity whose stated purpose nobody can name is already a finding.

Rewrite every line as an outcome with a number

An activity count is what the team did. An outcome is what happened to a developer because of it. The whole audit rests on refusing to confuse the two, so the rule is code rather than judgement.

export function reframe(line) {
  const context = `${line.count} ${line.unit}`
  if (!line.outcome) {
    return { activity: line.activity, text: `no outcome measured (${context})`, evidence: false }
  }
  const { label, value, target } = line.outcome
  return {
    activity: line.activity,
    text: `${label}: ${value} of ${target} (${context})`,
    evidence: true,
  }
}

The count survives in parentheses, as context. The number that leads the line is the outcome, and a line with no outcome says so instead of borrowing the count.

Score the same inventory both ways and the pitfall has a shape:

node pitfall.mjs

activity            by count  by outcome
Conference talks        3/3       0/3
Blog posts              3/3       1/3
Community forum         3/3       1/3
Sample apps             3/3       2/3
Office hours            3/3       2/3
Newsletter              3/3       3/3
Sponsored meetups       3/3       0/3
Changelog posts         3/3       2/3
average                 3.0       1.4

By count the program is perfect, because every planned thing was done. By outcome it is a 1.4. The left column is the review the team has been giving, accurate in every line but silent on what changed.

Days two to four: replace claims with evidence

Each stated purpose has a class of evidence that can contradict it, and the middle of the week is spent collecting it. Four classes cover most inventories.

Community response times come from the forum export: the share of question threads that got a first reply inside 24 hours, not the member count. Content-to-action attribution joins a source parameter on the link to the sign-up and request logs. A post is then credited with developers who made a first call within seven days, not with page views. Sample app health is whether each app builds green on CI against the current API version, which a status badge shows and Dependabot alerts sharpen. Event outcomes are the follow-ups: sign-ups naming the talk, or attendee problems closed within a week of the office hour.

Each outcome gets a target the team agrees to before the score is computed, and a score from how much of the target it reached:

export function reached(line) {
  if (!line.outcome) return 0
  const { value, target, better = 'higher' } = line.outcome
  return better === 'lower' ? target / value : value / target
}

/** 0 to 3, from how much of the target the outcome reached. */
export function scoreOutcome(line) {
  return Math.max(0, Math.min(3, Math.round(3 * reached(line))))
}

Where the evidence does not exist, the outcome stays empty. The line scores 0. Do not estimate. An audit that fills a gap with a plausible number has produced the slide it was meant to replace.

Day five: score the lines and present the changes

node demo.mjs

day five: Q3 scored

 0/3  Conference talks   sign-ups naming a talk in the referral field within 30 days: 9 of 60 (12 talks given)
 1/3  Blog posts         developers who made a first API call within 7 days of arriving from a post: 31 of 100 (24 posts published)
 1/3  Community forum    questions with a first reply inside 24 hours: 0.41 of 0.9 (1400 members)
 2/3  Sample apps        sample apps green on CI against the current API version: 3 of 6 (6 sample apps)
 2/3  Office hours       attendee problems closed within a week of the session: 35 of 58 (40 sessions held)
 3/3  Newsletter         sign-ups arriving from a newsletter link: 140 of 120 (12 issues sent)
 0/3  Sponsored meetups  no outcome measured (4 meetups sponsored)
 2/3  Changelog posts    community feature requests shipped this quarter: 4 of 6 (9 releases announced)

AAARRRP stage scores
  awareness    0/3 from 2 activities, 1 unmeasured
  acquisition  2/3 from 2 activities
  activation   2/3 from 1 activities
  retention    1.5/3 from 2 activities
  referral     no activity
  revenue      no activity
  product      2/3 from 1 activities

Orbit gravity added, ranked
  Office hours        114
  Community forum      84
  Conference talks     80
  Blog posts           50
  Changelog posts      25
  Newsletter            0
  Sample apps           0
  Sponsored meetups     0

three changes for next quarter
  - Sponsored meetups: measure an outcome next quarter, or stop (scored 0)
  - Conference talks: move "sign-ups naming a talk in the referral field within 30 days" from 9 to 60, or stop (scored 0)
  - Blog posts: move "developers who made a first API call within 7 days of arriving from a post" from 31 to 100, or stop (scored 1)

Read the three changes as the report’s argument. The unmeasured line comes first, whatever its count, because nothing about it can be discussed until it has a number. The other two are the measured lines furthest from their targets, and each change is the target itself: reach it next quarter, or stop.

Two stages read no activity, and that is a finding rather than a gap in the audit. Nobody is working on referral or revenue, and a funnel with two empty stages is a decision the team made without noticing.

Score the funnel and the gravity side by side

The two frames disagree, and the disagreement is the useful part. The funnel puts the newsletter first, at 3 of 3, because it beat its sign-up target. Orbit gives the newsletter zero, because a broadcast moves nobody inward. Office hours score 2 of 3 on the funnel and top the gravity table, because six Participants became Contributors and two Contributors became Advocates. The gravity of a move is weighted by the reach of the member who made it.³

export function gravity(line) {
  return line.orbit.reduce((sum, m) => sum + m.members * (love(m.to) - love(m.from)) * m.reach, 0)
}

Present both tables. A leadership team that funds the pipeline will read the funnel, and the gravity column is how you show them the office hours they were about to cut. Neither frame is the verdict on its own, and a report that shows one is choosing its conclusion.

Check it worked

Nine tests pin the rules. Two matter most. In this inventory, scoring by count never scores a line below scoring by outcome, and the newsletter and office hours change places between the two frames.

node --test audit.test.mjs

1..9
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 116.889862

The test runner is Node’s own node:test, so the audit runs anywhere the inventory file does. Run it again after every edit to inventory.json, and the literal Sponsored meetups: measure an outcome in the last test is the line to read: if it stops matching, an unmeasured activity has been given a number.

When it goes wrong

Every line scores 3. The targets were set after the numbers were known. Set targets on Monday, in writing, before any evidence is read.

The forum line scores 1 and the team says the forum is thriving. Both are true: 1,400 members is a count and 41% answered inside a day is the outcome. Present the outcome, and put the member count in the parentheses where the report keeps it.

The attribution is empty for every content line. The links carry no source parameter, so no post can be credited with anything. That is the first change for next quarter, ahead of any content decision, because a quarter without attribution cannot be audited at all.

An external audit lands in the same week and disagrees. It is scoring the questionnaire it brought, over 22 programs at once.⁴ Compare its line items with yours before comparing verdicts, and expect the lines it could not see to be the ones you scored lowest.

When not to do this

Do not run the audit in a week when the team cannot attend. Days two to four are spent pulling records the team owns, and a five-day audit done around them becomes a three-week audit done to them.

Do not present a single average. The 1.4 in the pitfall table is a rhetorical number, useful for one slide and wrong for every decision. A newsletter at 3 and a forum at 1 are not a program at 2. The report is the lines and the three changes.

Do not score against the Orbit frame if the inventory has no member trail. Gravity needs records of who moved, from which level, with what reach, and a team that has not kept them will invent them on day four. Score the funnel alone and add the member records to next quarter’s changes.

Do not commission an external audit to settle a dispute about the team and then run this one as well. Two reports with different line items, on the same desk in the same month, are read as the team marking its own homework. That is the accusation the external audit was hired to answer.

Related how-tos

• Choose developer experience metrics for your API

Last verified

Verified 2026-09-24 against Node 22.22.2. Every output block is what the command preceding it printed. The inventory is a constructed example, not a real team’s quarter. The two frameworks are cited for their stages and their formula and are not run as tools; the scoring rules are this page’s own.