How to evaluate a third-party SDK before you depend on it#

A pull request adds a vendor SDK, and the review approves it because the repository has eight thousand stars. Months later a bug in the SDK’s transport stalls your checkout. The fix was merged upstream weeks ago, but no release carries it. The review never asked how often the package ships.

What you get

You will end up with a script that scores each candidate from a dated snapshot of its registry and repository, and a test suite that pins every rule. This is for you if a vendor client is about to enter a service you operate.

Short answer

Before a vendor client enters your lockfile, record five facts about it in a dated snapshot. They are its license, its last five release dates, and the fixes merged since its last release. Add its open issues with their ages, and who answers when the vendor will not debug your case. Score every candidate, plain fetch included, against the same rows, and ignore the star count.

You will need

Node 22 or later and git, plus network access for the one script that collects the snapshot. Scoring reads a file and needs neither. Verified 2026-10-15 against Node 22.22.0, and the snapshot on this page was taken the same day: every number below describes 2026-10-15 and no other.

Two deeper checks belong to their own pages. Measure retry counts and tail latency with the retry and timeout audit. Whether the SDK lets you replace its HTTP layer is the subject of the transport-control page in this series.

Approaches compared

A well-maintained official SDK saves you code and costs you control. stripe-node has zero runtime dependencies and ships every two weeks, but a fix reaches you only if you track its majors. Octokit’s plugins cover pagination and throttling for you, at the price of eleven dependencies. A fetch client leaves the transport in your hands, and you write every layer on top of it.

Collect the facts on one day

A score is only comparable if every candidate is measured on the same day, so the facts go into a file named after that day. Release dates come from the npm registry’s time field, which npm view prints. Stars and the open count come from the deps.dev API, which mirrors GitHub’s counters.

The unreleased work comes from git itself. A clone that fetches commits without file contents is a few seconds per repository, and a log from the release tag to the default branch lists what has merged since.

function clone(repo) {
  const dir = join(work, repo.replace('/', '__'))
  // Commits and refs only. Neither trees nor file contents are needed to count commits.
  execFileSync('git', ['clone', '--quiet', '--bare', '--filter=tree:0', `https://github.com/${repo}`, dir])
  return dir
}

/** Commits on `head` that are not in `base`, oldest last, one per merged change. */
function commitsBetween(dir, base, head) {
  const out = git(dir, 'log', '--first-parent', '--format=%h%x09%cI%x09%s', `${base}..${head}`)
  return out ? out.split('\n').map((line) => {
    const [sha, at, ...subject] = line.split('\t')
    return { sha, date: new Date(at).toISOString().slice(0, 10), subject: subject.join('\t') }
  }) : []
}

--first-parent follows only the branch’s own line, so a repository that squash-merges yields one commit per merged pull request. Run the collector once, and commit what it writes:

node fetch-snapshot.mjs

A hand-written fetch client has no package to score, so score what it runs on. Node’s fetch is a bundled copy of undici, shipped inside each Node release.¹ Its release dates are Node’s, from the release index. The fixes it lacks are undici releases made after the copy inside the most recent Node 22.

Score every candidate against the same rows

The rule that matters most counts fixes, not commits. A dependency bump or a CI change on the default branch is not something you are waiting for.

export function unreleased(c, taken) {
  if (c.unreleased === null) {
    const why = c.lastTag ? `${c.lastTag} is not on ${c.branch}` : `no tag names release ${c.latest}`
    return { level: 'warning', text: `${why}, so unreleased work cannot be counted` }
  }
  const fixes = c.unreleased.filter((u) => isFix(u.subject))
  const where = c.bundled
    ? `${c.unreleased.length} commits in undici ${c.lastTag}..${c.upstreamTag}, not in Node ${c.latest}`
    : `${c.unreleased.length} commits on ${c.branch} since ${c.lastTag}`
  if (fixes.length === 0) return { level: 'ok', text: `${where}, 0 of them fixes`, fixes }
  const oldest = fixes.map((f) => f.date).sort()[0]
  const wait = daysBetween(oldest, taken)
  const text = `${where}, ${fixes.length} of them fixes, oldest ${wait} days`
  return { level: wait > LIMITS.unreleasedFixDays ? 'warning' : 'ok', text, fixes }
}

Every age is measured from the snapshot’s date, never from the clock. A score run next year gives the same answer about the same file.

node score.mjs snapshot-2026-10-15.json

snapshot taken 2026-10-15

stripe-node  (stripe 23.0.0, stripe/stripe-node)
  license          MIT, 0 runtime dependencies
  last 5 releases  23.0.0 2026-10-01, 22.6.2 2026-09-09, 22.6.1 2026-09-01, 22.6.0 2026-08-27, 22.5.0 2026-08-10
  cadence          last release 14 days before the snapshot, median gap 13 days
  unreleased       2 commits on master since v23.0.0, 0 of them fixes
  open issues      47 issues and pull requests; ages not collected
  support          fixes ship on the latest major only; first-time external pull requests are on hiatus, so the route is an issue
  verdict          adopt, once issues is checked

The support row is where stripe-node asks the most of you. Its versioning policy says earlier majors stay installable but receive no further updates.² Pin a major and you have pinned yourself out of security fixes, so budget an upgrade per major.

Score the code that runs, not the package you install

Stars rank the Octokit umbrella package first, while its last release is the oldest of the four. The transport it installs tells a different story.

node score.mjs snapshot-2026-10-15.json

Octokit  (octokit 5.0.5, octokit/octokit.js)
  license          MIT, 11 runtime dependencies
  last 5 releases  5.0.5 2025-10-31, 5.0.4 2025-10-17, 5.0.3 2025-05-27, 5.0.2 2025-05-20, 3.2.2 2025-05-20
  cadence          last release 349 days before the snapshot, median gap 11 days
  unreleased       14 commits on main since v5.0.5, 0 of them fixes
  open issues      52 issues and pull requests; ages not collected
  support          an issue with a reproducible test case; semantic-release ships every merged fix commit
  verdict          adopt with a plan, once issues is checked

Octokit transport  (@octokit/request 10.0.16, octokit/request.js)
  license          MIT, 6 runtime dependencies
  last 5 releases  10.0.16 2026-08-30, 10.0.15 2026-08-21, 10.0.14 2026-08-17, 10.0.13 2026-08-01, 10.0.12 2026-08-01
  cadence          last release 46 days before the snapshot, median gap 7 days
  unreleased       4 commits on main since v10.0.16, 0 of them fixes
  open issues      15 issues and pull requests; ages not collected
  support          the same process as the umbrella package; this is where the HTTP code lives
  verdict          adopt, once issues is checked

octokit 5.0.5 declares @octokit/core as ^7.0.6, so a fresh install resolves the core and request releases from August. The umbrella holds little code of its own, and the HTTP code lives in the packages it depends on. Octokit’s maintainer guide says its releases run through semantic-release, which publishes a patch for every merged commit typed fix under Conventional Commits. A merged fix there does not wait, which is why the zero means something.

The fetch client is the candidate with fixes it has not received:

node score.mjs snapshot-2026-10-15.json

fetch client  (Node 22.23.3, fetch is undici 6.28.1, nodejs/undici)
  license          MIT, 0 runtime dependencies
  last 5 releases  22.23.3 2026-09-23, 22.23.2 2026-07-28, 22.23.1 2026-06-22, 22.23.0 2026-06-17, 22.22.3 2026-05-13
  cadence          last release 22 days before the snapshot, median gap 36 days
  unreleased       11 commits in undici v6.28.1..v6.29.1, not in Node 22.23.3, 8 of them fixes, oldest 31 days
                     2026-10-15 fix(websocket): bound automatic PONG writes under backpressure
                     2026-10-15 fix(eventsource): bound retained SSE parser state
                     2026-10-15 fix(fetch): abort excessive content encoding responses
                     2026-10-15 fix: preserve Agent connection options
                     2026-10-15 fix: handle post-connect TLS certificate errors
                     2026-10-14 [v6.x] fix(fetch): fix aborting Streams (#3754) (#6000)
                     2026-09-17 Backport upgrade diagnostics lifecycle fixes to v6.x (#5833)
                     2026-09-14 fix(retry): settle exposed body on terminal failure (#5778)
  open issues      370 issues and pull requests; ages not collected
  support          you own the client; a fetch bug goes to undici and reaches you in a Node release
  verdict          adopt with a plan, once issues is checked

ranked by stars                    ranked by days since a release
  Octokit              7857         stripe-node          14
  fetch client         7710         fetch client         22
  stripe-node          4529         Octokit transport    46
  Octokit transport    265          Octokit              349

Plain fetch is the cheapest candidate to adopt, but the slowest to patch, because its transport ships inside Node. Two of those fixes change fetch itself, and they reach Node 22 only in a Node release that updates its bundled undici. The plan is to install undici from npm and pass its fetch, which puts its releases on your schedule.

The last block is the pitfall in two columns. Ranked by stars, the candidate that has not shipped for 349 days comes first and the package holding the HTTP code comes last.³

Check it worked

The suite pins each rule against a made-up candidate, then asserts two findings in the committed snapshot, so a refreshed snapshot that changes them fails loudly.

test('a fix merged 45 days ago and never released is a warning', () => {
  const c = { ...base, unreleased: [
    { sha: 'a1', date: '2026-08-31', subject: 'fix: retry on ECONNRESET' },
    { sha: 'a2', date: '2026-10-10', subject: 'chore(deps): bump vitest' },
  ] }
  const r = unreleased(c, TAKEN)
  assert.equal(r.level, 'warning')
  assert.match(r.text, /2 commits on main since v2\.0\.0, 1 of them fixes, oldest 45 days/)
  assert.equal(score(c, TAKEN).verdict, 'adopt with a plan, once issues is checked')
})

node --test score.test.mjs

1..15
# tests 15
# suites 0
# pass 15
# fail 0
# cancelled 0
# skipped 0
# todo 0

When it goes wrong

The unreleased row says the tag is not on the default branch. The project cuts releases from a release branch, so the tag is not an ancestor of main. Compare against that branch instead, or count from the merge base.

The open issues row says ages not collected. The run that produced this snapshot could not reach GitHub’s issues API, so the field is empty and the verdict names it as unchecked rather than passing it. A repository with 500 or more open issues gets the same answer, because a list cut at --limit cannot say how old its oldest issue is. With an authenticated gh, the collector fills it from this command:

gh issue list -R stripe/stripe-node --state open --limit 500 --json number,createdAt

A license reads SEE LICENSE IN LICENSE. The registry field points at a file, and the check rejects it because it cannot read the file. Read the file, then write the SPDX identifier you found into the snapshot by hand, with a note.

When not to do this

Do not rank candidates by stars. A star records that somebody once meant to come back, which says nothing about releases: in this snapshot the most starred package has the oldest release.

Do not score once and file the result. Every row moves, and the snapshot carries its date so that nobody mistakes last spring’s answer for today’s.

Do not adopt an official SDK for three endpoints. The SDK earns its dependencies when you call dozens of operations; for three, a fetch client and its tests are smaller than the audit.

Do not fork a package to get a fix that is merged but unreleased. Ask for a release in the issue tracker first, because a fork is a second package to score next year.

Do not let a clean score stand in for the two deeper checks. Retry defaults and transport control decide how the client behaves in an outage, which no registry field records.

Related how-tos

• Audit SDK retry and timeout defaults before adoption

• Judge whether an unofficial SDK is safe to depend on

• Check whether a vendor SDK lets you control the transport

Last verified

Verified 2026-10-15 against Node 22.22.0. Every output block is what the command preceding it printed. The snapshot was collected on 2026-10-15 by fetch-snapshot.mjs from the npm registry, deps.dev, nodejs.org, Node’s source tree for the bundled undici version, and commits-only clones of the four repositories. Issue ages were not collected.