How-to › Release and secure packages

How to run a scheduled live suite that gates an SDK release#

Run the integration tests of an SDK against a sandbox tenant every night, and refuse to publish a release unless the newest run is green and under 24 hours old.

Audience
API producer
Level
intermediate
Topic
Version, changelog and automate releases
Languages
TypeScript, Python and Go
Verified

The SDK’s tests pass against the mock on every pull request. The sandbox API has renamed a field since the mock was recorded, so the mock still answers the shape the tests expect. Nothing runs against the sandbox before release, so the first customer to call list gets a type error.

What you get

You will end up with a nightly workflow that runs the SDK’s integration suite against a sandbox tenant, and a release job that refuses to publish without a recent green run. This is for you if you publish an SDK and only mocks test it before release.

Short answer

Schedule a GitHub Actions workflow that runs the SDK’s integration suite against a sandbox tenant, with the key from the CI secret store and a sweep step that deletes tagged records. In the release workflow, read the runs with gh run list and publish only when the newest finished run succeeded within 24 hours. An sdkgen SDK reruns its mock corpus live when BOOKSHELF_TEST_LIVE=TRUE.

You will need

An SDK with an integration test suite, a sandbox tenant, and its key stored as a GitHub Actions secret. Node 22 or later runs the sample. The sample’s api.json is the five-operation Bookshelf document with a templated server, http://{sandbox}/v1, and an X-API-Key scheme. A stand-in on 127.0.0.1 plays the sandbox tenant.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Hand-written smoke tests in a workflow_dispatch jobA handful of calls you choose, run by hand before a releaseCoverage stops at the calls someone remembered to write, and nobody runs a manual job nightlyYou want the gate to need no person
NSwag client with smoke checksA TypeScript or C# client generated by NSwagNSwag writes no tests, so every live check is code you maintainYour generator ships a test suite
Schemathesis on a scheduleYou want drift between the spec and the deployed serverIt tests the server against the spec, so a bug inside the SDK passesThe SDK’s own code is what you need to prove
sdkgen live suiteAn sdkgen SDK, and the same tests offline and liveA failed delete inside the suite leaves the record it created behindYour sandbox cannot take writes
Swagger Codegen test stubsA Swagger Codegen client and a test file per API to fill inEvery stub body is pass, so the suite proves nothing until writtenYou need coverage on day one

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

The scheduled workflow and the release gate are the same whichever suite runs. What changes is what the suite covers. Reusing a generated corpus is coverage you did not write, at the price of fixtures you did not design. Schemathesis finds spec-versus-server drift but not SDK bugs.

Run the generated corpus against the sandbox

The generated entity test is one test with two modes. It reads BOOKSHELF_TEST_LIVE and, when it is TRUE, sends the same flow to the server instead of the in-memory mock:

    (0, node_test_1.test)('basic', async (t) => {
        const live = 'TRUE' === process.env.BOOKSHELF_TEST_LIVE;
        for (const op of ['create', 'list', 'update', 'load', 'remove']) {
            if (!live && (0, utility_1.maybeSkipControl)(t, 'entityOp', 'book.' + op, live))
                return;
        }
        const setup = basicSetup();

Offline, it runs against fixtures with fixed ids such as BOOK00:

node --test generated/dist-test/entity/book/BookEntity.test.js | tail -8
# tests 6
# suites 1
# pass 5
# fail 0
# cancelled 0
# skipped 1
# todo 0
# duration_ms 153.118268

Live, the run-a-live-suite how-to names the three inputs: BOOKSHELF_TEST_LIVE=TRUE, the key in BOOKSHELF_APIKEY, and each server variable as BOOKSHELF_SERVER_<NAME>. live.mjs starts the stand-in, sets all three, and runs the suite twice:

node live.mjs
CI secret matches the sandbox key
  live steps: 6 planned, 6 passed, 0 failed, 0 blocked
  suite exit: 0, green
  requests: POST /v1/books, GET /v1/books, PATCH /v1/books/bk1, GET /v1/books/bk1, DELETE /v1/books/bk1, GET /v1/books
  records left in the sandbox: 0
sandbox key rotated, CI secret not updated
  live steps: 6 planned, 0 passed, 3 failed, 3 blocked
  suite exit: 1, red
  requests: POST /v1/books, GET /v1/books, GET /v1/books
  records left in the sandbox: 0

The live flow used bk1, the id the server returned, rather than a fixture id. It deleted the record it made and then listed again to confirm the delete.

Schedule it and gate the release on it

The nightly workflow runs the suite with the secret and a repository variable for the host. Its sweep step runs with if: always(), so it runs after a red suite too:

on:
  schedule:
    - cron: '17 3 * * *'
  workflow_dispatch:

Pick a minute other than zero; GitHub documents delays at the start of every hour.1

The release workflow asks GitHub for the last runs and hands them to gate.mjs. The job declares actions: read, the permission the workflow runs API asks for, which gh run list calls:

        run: |
          gh run list --workflow live-suite.yml --branch main --limit 20 \
            --json status,conclusion,startedAt,event > runs.json
          node gate.mjs runs.json

The gate takes the newest finished run that was not cancelled or skipped, not the newest green one. A key rotated after a green run makes the next run red, and that red run is what blocks:

node gate.mjs runs/rotated.json 2026-11-20T09:00:00Z; echo "exit $?"
release blocked: newest live run is failure, 1.3 h ago
exit 1

runs/rotated.json holds a green nightly from 5.7 hours earlier. A rule of “any green run in the past 24 hours” would have published.

Write the live checks for Swagger Codegen and NSwag

Swagger Codegen writes test files you fill in. Its Python client came with test/test_default_api.py, one method per operation, each with a body of pass, so the scheduled job would be green from the first night:

    def test_list_books(self):
        """Test case for list_books

        """
        pass

Fill each stub with a call against configuration.host, which the client defaults to the document’s literal http://{sandbox}/v1. Run it as java -DapiTests=false -jar ... if you would rather not ship empty tests; the stub file is then left out.2

NSwag writes no tests and, for this document, no code for the X-API-Key scheme. Its TypeScript constructor takes a base URL and an object with a fetch, which is where the key goes:

  const fetchWithKey = (url, init = {}) => fetch(url, { ...init, headers: { ...init.headers, 'X-API-Key': key } })
  const client = new Client(`http://${host}/v1`, { fetch: fetchWithKey })

smoke.mjs is the hand-written smoke list over that client. It tags what it creates with live- and deletes it in a finally:

node smoke.mjs
list books: ok
create a tagged book: ok
load it back: ok
delete it: ok
records left: 0

Check it worked

Rotate the sandbox key without updating the secret, start the live workflow by hand, then push a release tag. The tests run that sequence against the stand-in. They also check the gate’s edges, the sweep, and the workflow shape:

node --test live.test.mjs | tail -8
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 21511.298524

live.mjs removes NODE_TEST_CONTEXT before it starts the inner node --test. With it inherited, the inner run reports to the outer one and exits 0 on a red suite.

When it goes wrong

The sandbox hits its record quota within a week. A delete failed inside the suite, and the suite does not retry it. cleanup.mjs makes the sandbox answer one DELETE with 500 for each suite:

node cleanup.mjs
generated live suite: red, POST bodies [{}]
  left after the suite: bk1 tagged=true
  sweep deleted 1, left after the sweep: none
smoke checks: delete it: 500
  left after the suite: bk1 tagged=true
  sweep deleted 1, left after the sweep: none

The generated flow builds its create body from the request schema’s required fields and examples. Bookshelf’s Book declares neither, so the POST carried {}. The update step writes the Mark01- value into author. A failed update alone does no harm, because the suite still runs its delete. A record whose update and delete both fail carries no mark, so give the suite a tenant of its own, or have the sweep delete everything in it.

The gate blocks every release, and the newest live run is weeks old. In a public repository GitHub disables a scheduled workflow after 60 days without repository activity, so no run is newer than the day it stopped. Re-enable the workflow; the gate failing closed is the intended behavior.

When not to do this

Do not point a live suite at production. The generated flow creates, updates, and deletes, and a failed delete leaves a record behind, as cleanup.mjs shows.

Do not gate on a run that tested other code. A nightly run checks main as it stood that night, so a release cut from a later commit needs a manual run first.

Do not adopt sdkgen for its live suite alone. If your SDK comes from Swagger Codegen, filling its stubs keeps one generator and one test layout; with NSwag, a dozen smoke checks over its client cover the calls customers make. Run Schemathesis beside either when the risk is the server drifting from the spec.

Last verified

Verified 2026-11-20 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, NSwag 14.8.0 and Swagger Codegen 3.0.85. Every output block is what the command preceding it printed. NSwag and Swagger Codegen were run once on api.json and their output committed. The two workflows were not run on GitHub; a test checks their shape. Schemathesis was not run.

Footnotes

  1. GitHub’s schedule event notes that runs can be delayed during periods of high load, and names the start of every hour as such a period. It adds that some queued jobs may be dropped if the load is high enough. Every schedule written 0 * * * * makes the hour a little busier. ↩︎ Back to text

  2. Swagger Codegen’s selective generation page describes -DapiTests and -DmodelTests as options that default to true. They are Java system properties rather than generator flags, which is why they go before -jar. The generated class names itself in its docstring as DefaultApi unit test stubs. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.