# How to evaluate a third-party SDK before you depend on it > Score each candidate client on license, release cadence, unreleased fixes, open issues, and support, from a dated snapshot instead of its star count. Source: https://voxgig.com/howto/evaluate-a-third-party-sdk - Audience: api-consumer - Level: intermediate - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Before a vendor client enters your lockfile, record five facts about it in a dated snapshot. They are its license, its last five release dates, and the fixes merged since its last release. Add its open issues with their ages, and who answers when the vendor will not debug your case. Score every candidate, plain fetch included, against the same rows, and ignore the star count. --- ## You will need Node 22 or later and `git`, plus network access for the one script that collects the snapshot. Scoring reads a file and needs neither. Verified 2026-10-15 against Node 22.22.0, and the snapshot on this page was taken the same day: every number below describes 2026-10-15 and no other. Two deeper checks belong to their own pages. Measure retry counts and tail latency with [the retry and timeout audit](/howto/audit-sdk-retry-and-timeout-defaults). Whether the SDK lets you replace its HTTP layer is the subject of the transport-control page in this series. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [A hand-written fetch client](https://developer.mozilla.org/en-US/docs/Web/API/Window/fetch) | A handful of operations, and you want the transport in your own hands | Every operation, error type and retry is your code, and transport fixes arrive only in Node releases | You call dozens of operations, or the API changes every month | | [Octokit](https://octokit.github.io/rest.js/v22/) | You call GitHub's REST or GraphQL APIs and want typed methods and plugins | Eleven runtime dependencies, and a release history split across a dozen repositories you have to read | You call three endpoints, where `@octokit/request` alone is less to audit | | [stripe-node](https://docs.stripe.com/sdks) | You call Stripe and want typed resources, idempotency keys and retries built in | Fixes land on the highest major only, so falling one major behind means no fixes at all | Your platform needs a transport that its `httpClient` option cannot express | A well-maintained official SDK saves you code and costs you control. stripe-node has zero runtime dependencies and ships every two weeks, but a fix reaches you only if you track its majors. Octokit's plugins cover pagination and throttling for you, at the price of eleven dependencies. A fetch client leaves the transport in your hands, and you write every layer on top of it. ## Collect the facts on one day A score is only comparable if every candidate is measured on the same day, so the facts go into a file named after that day. Release dates come from the npm registry's `time` field, which [npm view](https://docs.npmjs.com/cli/v10/commands/npm-view/) prints. Stars and the open count come from the [deps.dev API](https://docs.deps.dev/api/v3/), which mirrors GitHub's counters. The unreleased work comes from `git` itself. A clone that fetches commits without file contents is a few seconds per repository, and a log from the release tag to the default branch lists what has merged since. ```js title="fetch-snapshot.mjs" function clone(repo) { const dir = join(work, repo.replace('/', '__')) // Commits and refs only. Neither trees nor file contents are needed to count commits. execFileSync('git', ['clone', '--quiet', '--bare', '--filter=tree:0', `https://github.com/${repo}`, dir]) return dir } /** Commits on `head` that are not in `base`, oldest last, one per merged change. */ function commitsBetween(dir, base, head) { const out = git(dir, 'log', '--first-parent', '--format=%h%x09%cI%x09%s', `${base}..${head}`) return out ? out.split('\n').map((line) => { const [sha, at, ...subject] = line.split('\t') return { sha, date: new Date(at).toISOString().slice(0, 10), subject: subject.join('\t') } }) : [] } ``` [`--first-parent`](https://git-scm.com/docs/git-log) follows only the branch's own line, so a repository that squash-merges yields one commit per merged pull request. Run the collector once, and commit what it writes: ```bash node fetch-snapshot.mjs ``` A hand-written fetch client has no package to score, so score what it runs on. Node's `fetch` is a bundled copy of [undici](https://undici.nodejs.org/), shipped inside each Node release.[^1] Its release dates are Node's, from [the release index](https://nodejs.org/dist/index.json). The fixes it lacks are undici releases made after the copy inside the most recent Node 22. ## Score every candidate against the same rows The rule that matters most counts fixes, not commits. A dependency bump or a CI change on the default branch is not something you are waiting for. ```js title="checks.mjs" export function unreleased(c, taken) { if (c.unreleased === null) { const why = c.lastTag ? `${c.lastTag} is not on ${c.branch}` : `no tag names release ${c.latest}` return { level: 'warning', text: `${why}, so unreleased work cannot be counted` } } const fixes = c.unreleased.filter((u) => isFix(u.subject)) const where = c.bundled ? `${c.unreleased.length} commits in undici ${c.lastTag}..${c.upstreamTag}, not in Node ${c.latest}` : `${c.unreleased.length} commits on ${c.branch} since ${c.lastTag}` if (fixes.length === 0) return { level: 'ok', text: `${where}, 0 of them fixes`, fixes } const oldest = fixes.map((f) => f.date).sort()[0] const wait = daysBetween(oldest, taken) const text = `${where}, ${fixes.length} of them fixes, oldest ${wait} days` return { level: wait > LIMITS.unreleasedFixDays ? 'warning' : 'ok', text, fixes } } ``` Every age is measured from the snapshot's date, never from the clock. A score run next year gives the same answer about the same file. ```bash node score.mjs snapshot-2026-10-15.json ``` ```text output snapshot taken 2026-10-15 stripe-node (stripe 23.0.0, stripe/stripe-node) license MIT, 0 runtime dependencies last 5 releases 23.0.0 2026-10-01, 22.6.2 2026-09-09, 22.6.1 2026-09-01, 22.6.0 2026-08-27, 22.5.0 2026-08-10 cadence last release 14 days before the snapshot, median gap 13 days unreleased 2 commits on master since v23.0.0, 0 of them fixes open issues 47 issues and pull requests; ages not collected support fixes ship on the latest major only; first-time external pull requests are on hiatus, so the route is an issue verdict adopt, once issues is checked ``` The support row is where stripe-node asks the most of you. Its [versioning policy](https://docs.stripe.com/sdks/versioning?lang=node) says earlier majors stay installable but receive no further updates.[^2] Pin a major and you have pinned yourself out of security fixes, so budget an upgrade per major. ## Score the code that runs, not the package you install Stars rank the Octokit umbrella package first, while its last release is the oldest of the four. The transport it installs tells a different story. ```bash node score.mjs snapshot-2026-10-15.json ``` ```text output Octokit (octokit 5.0.5, octokit/octokit.js) license MIT, 11 runtime dependencies last 5 releases 5.0.5 2025-10-31, 5.0.4 2025-10-17, 5.0.3 2025-05-27, 5.0.2 2025-05-20, 3.2.2 2025-05-20 cadence last release 349 days before the snapshot, median gap 11 days unreleased 14 commits on main since v5.0.5, 0 of them fixes open issues 52 issues and pull requests; ages not collected support an issue with a reproducible test case; semantic-release ships every merged fix commit verdict adopt with a plan, once issues is checked Octokit transport (@octokit/request 10.0.16, octokit/request.js) license MIT, 6 runtime dependencies last 5 releases 10.0.16 2026-08-30, 10.0.15 2026-08-21, 10.0.14 2026-08-17, 10.0.13 2026-08-01, 10.0.12 2026-08-01 cadence last release 46 days before the snapshot, median gap 7 days unreleased 4 commits on main since v10.0.16, 0 of them fixes open issues 15 issues and pull requests; ages not collected support the same process as the umbrella package; this is where the HTTP code lives verdict adopt, once issues is checked ``` `octokit` 5.0.5 declares `@octokit/core` as `^7.0.6`, so a fresh install resolves the core and request releases from August. The umbrella holds little code of its own, and the HTTP code lives in the packages it depends on. Octokit's [maintainer guide](https://github.com/octokit/octokit.js/blob/main/MAINTAINING.md) says its releases run through [semantic-release](https://semantic-release.gitbook.io/semantic-release), which publishes a patch for every merged commit typed `fix` under [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). A merged fix there does not wait, which is why the zero means something. The fetch client is the candidate with fixes it has not received: ```bash node score.mjs snapshot-2026-10-15.json ``` ```text output fetch client (Node 22.23.3, fetch is undici 6.28.1, nodejs/undici) license MIT, 0 runtime dependencies last 5 releases 22.23.3 2026-09-23, 22.23.2 2026-07-28, 22.23.1 2026-06-22, 22.23.0 2026-06-17, 22.22.3 2026-05-13 cadence last release 22 days before the snapshot, median gap 36 days unreleased 11 commits in undici v6.28.1..v6.29.1, not in Node 22.23.3, 8 of them fixes, oldest 31 days 2026-10-15 fix(websocket): bound automatic PONG writes under backpressure 2026-10-15 fix(eventsource): bound retained SSE parser state 2026-10-15 fix(fetch): abort excessive content encoding responses 2026-10-15 fix: preserve Agent connection options 2026-10-15 fix: handle post-connect TLS certificate errors 2026-10-14 [v6.x] fix(fetch): fix aborting Streams (#3754) (#6000) 2026-09-17 Backport upgrade diagnostics lifecycle fixes to v6.x (#5833) 2026-09-14 fix(retry): settle exposed body on terminal failure (#5778) open issues 370 issues and pull requests; ages not collected support you own the client; a fetch bug goes to undici and reaches you in a Node release verdict adopt with a plan, once issues is checked ranked by stars ranked by days since a release Octokit 7857 stripe-node 14 fetch client 7710 fetch client 22 stripe-node 4529 Octokit transport 46 Octokit transport 265 Octokit 349 ``` Plain fetch is the cheapest candidate to adopt, but the slowest to patch, because its transport ships inside Node. Two of those fixes change `fetch` itself, and they reach Node 22 only in a Node release that updates its bundled undici. The plan is to install undici from npm and pass its `fetch`, which puts its releases on your schedule. The last block is the pitfall in two columns. Ranked by stars, the candidate that has not shipped for 349 days comes first and the package holding the HTTP code comes last.[^3] ## Check it worked The suite pins each rule against a made-up candidate, then asserts two findings in the committed snapshot, so a refreshed snapshot that changes them fails loudly. ```js title="score.test.mjs" test('a fix merged 45 days ago and never released is a warning', () => { const c = { ...base, unreleased: [ { sha: 'a1', date: '2026-08-31', subject: 'fix: retry on ECONNRESET' }, { sha: 'a2', date: '2026-10-10', subject: 'chore(deps): bump vitest' }, ] } const r = unreleased(c, TAKEN) assert.equal(r.level, 'warning') assert.match(r.text, /2 commits on main since v2\.0\.0, 1 of them fixes, oldest 45 days/) assert.equal(score(c, TAKEN).verdict, 'adopt with a plan, once issues is checked') }) ``` ```bash node --test score.test.mjs ``` ```text output 1..15 # tests 15 # suites 0 # pass 15 # fail 0 # cancelled 0 # skipped 0 # todo 0 ``` ## When it goes wrong The unreleased row says the tag is not on the default branch. The project cuts releases from a release branch, so the tag is not an ancestor of `main`. Compare against that branch instead, or count from the merge base. The open issues row says ages not collected. The run that produced this snapshot could not reach GitHub's issues API, so the field is empty and the verdict names it as unchecked rather than passing it. A repository with 500 or more open issues gets the same answer, because a list cut at `--limit` cannot say how old its oldest issue is. With an authenticated [gh](https://cli.github.com/manual/gh_issue_list), the collector fills it from this command: ```bash gh issue list -R stripe/stripe-node --state open --limit 500 --json number,createdAt ``` A license reads `SEE LICENSE IN LICENSE`. The registry field points at a file, and the check rejects it because it cannot read the file. Read the file, then write the [SPDX identifier](https://spdx.org/licenses/) you found into the snapshot by hand, with a note. ## When not to do this Do not rank candidates by stars. A star records that somebody once meant to come back, which says nothing about releases: in this snapshot the most starred package has the oldest release. Do not score once and file the result. Every row moves, and the snapshot carries its date so that nobody mistakes last spring's answer for today's. Do not adopt an official SDK for three endpoints. The SDK earns its dependencies when you call dozens of operations; for three, a fetch client and its tests are smaller than the audit. Do not fork a package to get a fix that is merged but unreleased. Ask for a release in the issue tracker first, because a fork is a second package to score next year. Do not let a clean score stand in for the two deeper checks. Retry defaults and transport control decide how the client behaves in an outage, which no registry field records. ## Related how-tos - [Audit SDK retry and timeout defaults before adoption](/howto/audit-sdk-retry-and-timeout-defaults) - [Judge whether an unofficial SDK is safe to depend on](/howto/judge-an-unofficial-sdk-before-adoption) - [Check whether a vendor SDK lets you control the transport](/howto/check-transport-control-in-a-vendor-sdk) ## Last verified Verified 2026-10-15 against Node 22.22.0. Every output block is what the command preceding it printed. The snapshot was collected on 2026-10-15 by `fetch-snapshot.mjs` from the npm registry, deps.dev, nodejs.org, Node's source tree for the bundled undici version, and commits-only clones of the four repositories. Issue ages were not collected. [^1]: The project's documentation explains the name in one line of arithmetic: HTTP/1.1, then 11, then eleven, then [undici](https://undici.nodejs.org/), which is Italian for it. It adds that the name is also a Stranger Things reference, which gives the transport under every `fetch` call in Node two etymologies and one version number. [^2]: The same [policy page](https://docs.stripe.com/sdks/versioning?lang=node) dates the runtime end as well. Node 22 reaches end of life in April 2027, Stripe keeps supporting it through its September 2027 and March 2028 API releases, and the September 2028 major drops it. The SDK's support for the runtime outlasts the runtime's own by seventeen months. [^3]: GitHub's [documentation on stars](https://docs.github.com/en/get-started/exploring-projects-on-github/saving-repositories-with-stars) describes starring as a way to keep track of projects and to find a repository again later. It adds that a star shows appreciation to the maintainer, and that many of GitHub's repository rankings depend on the count. A bookmark that doubles as a ranking measures how many people intended to return.