# How to publish hundreds of generated SDK repos from one pipeline

> Shard a release across hundreds of generated SDK repositories, isolate each failure, pace publishes under a registry limit, and record each result on a status page.

Source: https://voxgig.com/howto/publish-hundreds-of-generated-sdk-repos

- Audience: platform-team
- Level: advanced
- Languages: typescript, python, go, php, ruby, lua
- Verified: 2026-11-22
- Published: 2026-11-22

## Short answer

Sort the repositories and give each CI job one shard of the list. In each shard, check the package's name, version, and disclaimer, ask the registry whether those bytes are already there, and publish under a pace that divides the registry budget between shards. Record every repository as published, skipped, or failed, and keep going. A rerun then publishes only what is missing.

---
## You will need

Node 22 or later with npm 10, and a list of generated repositories that each hold a package. Run
[Release six SDK packages from one commit](/howto/release-six-sdk-packages-from-one-commit) and
[Recover when one registry upload fails mid release](/howto/recover-from-a-partial-multi-registry-release)
first: this page repeats their per-repository steps hundreds of times. The sample publishes to a
registry stand-in on 127.0.0.1 with the real `npm publish`, and needs no account. Only the
TypeScript package of each fixture is published: the other five languages appear in the registry
scan alone.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [GitHub Actions reusable workflows with multi-gitter](https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows) | Each repository releases itself, and one change must reach every repository | A pull request per repository to roll out, and no single view of what released | One pipeline should own the release |
| [Kiota](https://learn.microsoft.com/en-us/openapi/kiota/using) | Many clients regenerated from their lock files with `kiota update` | No publishing at all, and `kiota update` 1.38.0 exited 0 over an unreadable description | You need the tool to report a failed client |
| [OpenAPI Generator batch](https://openapi-generator.tech/docs/usage/) | Many clients generated in one run from one config per client | Generates only; publishing and status are still your loop | Your clients come from different generators |
| [per-language monorepos as in the Azure and AWS SDKs](https://github.com/Azure/azure-sdk-for-js) | You own the APIs and want one repository per language | Each client's history, issues, and releases share one repository with every other service | Each client has its own upstream and its own cadence |
| [SDK Catalog fan-out](https://voxgig.com/voxgig-sdk) | Hundreds of generated repositories, one per API, each with several language targets | One repository per API to keep; the scan found Go modules for 636 and two npm packages | A handful of SDKs that fit in one repository |

Voxgig maintains the SDK Catalog, which is the subject of this page. Kiota, OpenAPI Generator,
multi-gitter, and the two monorepos are other projects'.

A monorepo such as [aws-sdk-js-v3](https://github.com/aws/aws-sdk-js-v3), with a separate package
per service, keeps every release in one place and ties them together. A fan-out keeps each repository
separate, so isolation is cheap and the status page has to be built. multi-gitter changes many
repositories but publishes none of them itself.

## Know what the catalog has published

`record.sh` asked each registry about every repository the site lists. The record is one day's
reading:

```bash
cat records/registries.txt
```

```text output
scanned on 2026-11-22
repositories the site lists: 637
npm: 2 (learnworlds, voxgig-solardemo)
PyPI: 0
RubyGems: 0
Go module proxy: 636
Packagist vendor voxgig-sdk: 0 packages
LuaRocks user voxgig-sdk: status 404
```

The [Go module proxy](https://proxy.golang.org/) serves a tagged module straight from its
repository, with no upload step, which is why 636 resolve. The two npm packages use the short name
without `-sdk`. No catalog package is on PyPI, Packagist, RubyGems, or LuaRocks.

## Shard the list and isolate each repository

Sort the list, so shard *k* of *n* is the same repositories on every machine:

```js title="fanout.mjs"
export function shard(repos, index, total) {
  if (!Number.isInteger(total) || total < 1 || !Number.isInteger(index) || index < 1 || index > total) {
    throw new Error(`bad shard ${index}/${total}`)
  }
  return repos.filter((_, i) => i % total === index - 1)
}
```

GitHub Actions caps a job matrix at 256 jobs per run and a hosted job at six hours, as its
[limits page](https://docs.github.com/en/actions/reference/limits) states,[^1] so 637 repositories
need fewer shards than repositories. `publishShard` catches every error per repository and records
it. `preflight` refuses a package whose name is outside `@voxgig-sdk/`, and one whose README
does not match `UNAFFILIATED`, the pattern for the phrase not affiliated with.

Split the registry's budget between the shards, with headroom for npm's start-up time:

```js title="demo.mjs"
const SHARDS = 3
const BUDGET = { limit: 3, windowMs: 1000 } // the stand-in's limit, shared by every shard
// Each shard takes a third of the budget, over a window half again as long, so
// the time npm takes to start cannot bunch two publishes into one window.
const PER_SHARD = { limit: Math.max(1, Math.floor(BUDGET.limit / SHARDS)), windowMs: BUDGET.windowMs * 1.5 }
```

The limit is the stand-in's own, because npm documents no publish rate. A 429 that gets past the
pace is retried after a growing wait of the loop's own, not the `Retry-After` value that
[RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html#section-4) lets a server send.

## Run the fan-out twice

`demo.mjs` makes nine fixture repositories named after catalog entries, cuts one manifest short,
strips one README of its disclaimer, and runs three shards. Then it restores the README and
runs again:

```bash
node demo.mjs
```

```text output
run 1
| repository | shard | package | result | note |
| --- | --- | --- | --- | --- |
| abhi-sdk | 1/3 | @voxgig-sdk/abhi-sdk@0.0.1 | published |  |
| california-state-portal-sdk | 2/3 | @voxgig-sdk/california-state-portal-sdk@0.0.1 | published |  |
| domain-whois-sdk | 3/3 | @voxgig-sdk/domain-whois-sdk@0.0.1 | published |  |
| free-uv-index-sdk | 1/3 | @voxgig-sdk/free-uv-index-sdk@0.0.1 | published |  |
| ip-intelligence-api2-sdk | 2/3 | @voxgig-sdk/ip-intelligence-api2-sdk@0.0.1 | published |  |
| opensanctum-sdk | 3/3 | - | failed | package.json is not valid JSON |
| rick-and-morty-sdk | 1/3 | @voxgig-sdk/rick-and-morty-sdk@0.0.1 | published |  |
| top-news-sdk | 2/3 | - | failed | README does not say the client is unaffiliated |
| yugi-limit-regulation-sdk | 3/3 | @voxgig-sdk/yugi-limit-regulation-sdk@0.0.1 | published |  |

published 7, skipped 0, failed 2

run 2, after the top-news-sdk README is fixed
| repository | shard | package | result | note |
| --- | --- | --- | --- | --- |
| abhi-sdk | 1/3 | @voxgig-sdk/abhi-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| california-state-portal-sdk | 2/3 | @voxgig-sdk/california-state-portal-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| domain-whois-sdk | 3/3 | @voxgig-sdk/domain-whois-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| free-uv-index-sdk | 1/3 | @voxgig-sdk/free-uv-index-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| ip-intelligence-api2-sdk | 2/3 | @voxgig-sdk/ip-intelligence-api2-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| opensanctum-sdk | 3/3 | - | failed | package.json is not valid JSON |
| rick-and-morty-sdk | 1/3 | @voxgig-sdk/rick-and-morty-sdk@0.0.1 | skipped | already on the registry with the same bytes |
| top-news-sdk | 2/3 | @voxgig-sdk/top-news-sdk@0.0.1 | published |  |
| yugi-limit-regulation-sdk | 3/3 | @voxgig-sdk/yugi-limit-regulation-sdk@0.0.1 | skipped | already on the registry with the same bytes |

published 1, skipped 7, failed 1

registry: 8 packages, 8 at 0.0.1
```

The broken manifest failed alone, and its shard went on to `yugi-limit-regulation-sdk`. The
second run compared each package's `npm pack` integrity with the registry's and skipped the seven
that matched. Commit the table as the status page, because a registry lists what arrived and
says nothing about what failed.

## Fan out with Kiota and OpenAPI Generator

Neither tool publishes, so each team keeps a loop like `publishShard` and changes only the
generate step. `record.sh` gave both three clients, one from a description cut off mid-file:

```bash
cat records/generators.txt
```

```text output
OpenAPI Generator 7.27.0, batch over three configs, broken.json first
batch:
  exit 1, [FAIL] Completed with 1 failures, 2 successes
  written: quotes-sdk tides-sdk
batch --fail-fast:
  exit 1, no summary line
  written: 
Kiota 1.38.0, three clients generated, then broken.json cut off and kiota update -o repos
  exit 0
  fail: Kiota.Builder.KiotaBuilder[49154641]
  Update of 3 clients completed successfully
  OpenAPI error reported: 1
  broken-sdk files the update changed: repos/broken-sdk/ts/.kiota.log repos/broken-sdk/ts/kiota-lock.json
  broken-sdk files before and after: 4 and 5
```

OpenAPI Generator's `batch` isolates the bad config unless you pass `--fail-fast`, and its exit
status says one failed. Kiota's
[`kiota update`](https://learn.microsoft.com/en-us/openapi/kiota/using) refreshes every client
under a folder from its lock file, and here it reported success over a client it could not
regenerate. It rewrote that client's lock file and added a log, and left its code alone. Grep its output for `fail:` per client, and do not trust the exit status of that
release.

## Check it worked

```bash
node --test fanout.test.mjs
```

```text output
1..12
# tests 12
# suites 0
# pass 12
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

The test that carries the page asserts the two status tables, and that the registry holds eight
packages and not `opensanctum-sdk`.

## When it goes wrong

A rerun fails on everything already published.
[`npm publish`](https://docs.npmjs.com/cli/v10/commands/npm-publish/) fails when the name and
version exist. Read the registry before publishing, and treat a refusal as published only when
the integrity matches, which `publishShard` does after an E403.

A regenerated package keeps its version. Its bytes then differ from the registry's, and a test
shows `publishShard` failing it rather than skipping it. Bump the version in the generator.

A package publishes somewhere else. An `.npmrc` inside the package can name a registry for the
`@voxgig-sdk` scope, which beats `--registry`, so `publishShard` passes the scoped registry too and
`preflight` refuses a package that carries one. A test shows nothing reaching a second registry.

The shards hit 429 together. Each shard paced itself to the full budget. Divide the budget by the
shard count, as `PER_SHARD` does.

## When not to do this

Do not fan out a handful of SDKs. Below a few dozen repositories, the per-language monorepo or a
loop in one job gives one release history and needs no status page.

Do not publish under a name that implies the API owner made the client. The catalog scopes every
package under `@voxgig-sdk/` and says in its READMEs that it is not affiliated, and `preflight`
refuses a package that does neither.

Do not treat the SDK Catalog's fan-out as proven for Python, PHP, Ruby, or Lua: none of those
registries holds a catalog package. Prefer OpenAPI Generator's `batch` when one tool generates
every client, because it reports failures in its exit status. Prefer Kiota when the clients already
carry lock files, so one command refreshes them all.

## Related how-tos

- [Release six SDK packages from one commit](/howto/release-six-sdk-packages-from-one-commit)

- [Recover when one registry upload fails mid release](/howto/recover-from-a-partial-multi-registry-release)

- [Choose a release automation tool for an SDK repository](/howto/choose-a-release-automation-tool-for-an-sdk)

- [Build a CI matrix for an SDK across six languages](/howto/sdk-ci-matrix-across-six-languages)

- [Judge whether an unofficial SDK is safe to depend on](/howto/judge-an-unofficial-sdk-before-adoption)

- [Tag a polyglot SDK repo so Go modules resolve](/howto/tag-a-polyglot-repo-for-go-modules)

## Last verified

Verified 2026-11-22 against Node 22.23.5 and npm 10.9.9, publishing to a stand-in rather than to
npm. `record.sh` queried npm, PyPI, RubyGems, Packagist, LuaRocks, and the Go module proxy for all
637 listed repositories, and ran OpenAPI Generator 7.27.0 and Kiota 1.38.0. Kiota ran once, with its
telemetry left on, and was not run again; its 1.39.0 release was not tested. A second run of
everything else in `record.sh` gave the same records.

[^1]: The same [limits page](https://docs.github.com/en/actions/reference/limits) gives a workflow run 35 days in all. A release of 637 repositories is
therefore bounded by the six hours each job gets rather than by the run. GitHub states both limits for
hosted runners and gives a self-hosted job five days.