# How to run a scheduled live suite that gates an SDK release

> Run the integration tests of an SDK against a sandbox tenant every night, and refuse to publish a release unless the newest run is green and under 24 hours old.

Source: https://voxgig.com/howto/scheduled-live-suite-before-an-sdk-release

- Audience: api-producer
- Level: intermediate
- Languages: typescript, python, go
- Verified: 2026-11-20
- Published: 2026-11-20

## Short answer

Schedule a GitHub Actions workflow that runs the SDK's integration suite against a sandbox tenant, with the key from the CI secret store and a sweep step that deletes tagged records. In the release workflow, read the runs with gh run list and publish only when the newest finished run succeeded within 24 hours. An sdkgen SDK reruns its mock corpus live when BOOKSHELF_TEST_LIVE=TRUE.

---
## You will need

An SDK with an integration test suite, a sandbox tenant, and its key stored as a
[GitHub Actions secret](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets).
Node 22 or later runs the sample. The sample's `api.json` is the five-operation Bookshelf
document with a templated server, `http://{sandbox}/v1`, and an `X-API-Key` scheme. A stand-in
on 127.0.0.1 plays the sandbox tenant.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Hand-written smoke tests in a `workflow_dispatch` job](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows) | A handful of calls you choose, run by hand before a release | Coverage stops at the calls someone remembered to write, and nobody runs a manual job nightly | You want the gate to need no person |
| [NSwag client with smoke checks](https://github.com/RicoSuter/NSwag) | A TypeScript or C# client generated by NSwag | NSwag writes no tests, so every live check is code you maintain | Your generator ships a test suite |
| [Schemathesis on a schedule](https://schemathesis.readthedocs.io/en/stable/quick-start/) | You want drift between the spec and the deployed server | It tests the server against the spec, so a bug inside the SDK passes | The SDK's own code is what you need to prove |
| [sdkgen live suite](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/run-a-live-suite.md) | An sdkgen SDK, and the same tests offline and live | A failed delete inside the suite leaves the record it created behind | Your sandbox cannot take writes |
| [Swagger Codegen test stubs](https://github.com/swagger-api/swagger-codegen) | A Swagger Codegen client and a test file per API to fill in | Every stub body is `pass`, so the suite proves nothing until written | You need coverage on day one |

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

The scheduled workflow and the release gate are the same whichever suite runs. What changes is
what the suite covers. Reusing a generated corpus is coverage you did not write, at the price of
fixtures you did not design. Schemathesis finds spec-versus-server drift but not SDK bugs.

## Run the generated corpus against the sandbox

The generated entity test is one test with two modes. It reads `BOOKSHELF_TEST_LIVE` and, when
it is `TRUE`, sends the same flow to the server instead of the in-memory mock:

```js title="generated/dist-test/entity/book/BookEntity.test.js"
    (0, node_test_1.test)('basic', async (t) => {
        const live = 'TRUE' === process.env.BOOKSHELF_TEST_LIVE;
        for (const op of ['create', 'list', 'update', 'load', 'remove']) {
            if (!live && (0, utility_1.maybeSkipControl)(t, 'entityOp', 'book.' + op, live))
                return;
        }
        const setup = basicSetup();
```

Offline, it runs against fixtures with fixed ids such as `BOOK00`:

```bash
node --test generated/dist-test/entity/book/BookEntity.test.js | tail -8
```

```text output
# tests 6
# suites 1
# pass 5
# fail 0
# cancelled 0
# skipped 1
# todo 0
# duration_ms 153.118268
```

Live, the [run-a-live-suite how-to](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/run-a-live-suite.md)
names the three inputs: `BOOKSHELF_TEST_LIVE=TRUE`, the key in `BOOKSHELF_APIKEY`, and each
server variable as `BOOKSHELF_SERVER_<NAME>`. `live.mjs` starts the stand-in, sets all three,
and runs the suite twice:

```bash
node live.mjs
```

```text output
CI secret matches the sandbox key
  live steps: 6 planned, 6 passed, 0 failed, 0 blocked
  suite exit: 0, green
  requests: POST /v1/books, GET /v1/books, PATCH /v1/books/bk1, GET /v1/books/bk1, DELETE /v1/books/bk1, GET /v1/books
  records left in the sandbox: 0
sandbox key rotated, CI secret not updated
  live steps: 6 planned, 0 passed, 3 failed, 3 blocked
  suite exit: 1, red
  requests: POST /v1/books, GET /v1/books, GET /v1/books
  records left in the sandbox: 0
```

The live flow used `bk1`, the id the server returned, rather than a fixture id. It deleted the
record it made and then listed again to confirm the delete.

## Schedule it and gate the release on it

The nightly workflow runs the suite with the secret and a repository variable for the host. Its
sweep step runs with `if: always()`, so it runs after a red suite too:

```yaml title="workflows/live-suite.yml"
on:
  schedule:
    - cron: '17 3 * * *'
  workflow_dispatch:
```

Pick a minute other than zero; GitHub documents delays at the start of every hour.[^1]

The release workflow asks GitHub for the last runs and hands them to `gate.mjs`. The job
declares `actions: read`, the permission the
[workflow runs API](https://docs.github.com/en/rest/actions/workflow-runs) asks for, which
[`gh run list`](https://cli.github.com/manual/gh_run_list) calls:

```yaml title="workflows/release.yml"
        run: |
          gh run list --workflow live-suite.yml --branch main --limit 20 \
            --json status,conclusion,startedAt,event > runs.json
          node gate.mjs runs.json
```

The gate takes the newest finished run that was not cancelled or skipped, not the newest green one. A key rotated after a green
run makes the next run red, and that red run is what blocks:

```bash
node gate.mjs runs/rotated.json 2026-11-20T09:00:00Z; echo "exit $?"
```

```text output
release blocked: newest live run is failure, 1.3 h ago
exit 1
```

`runs/rotated.json` holds a green nightly from 5.7 hours earlier. A rule of "any green run in the
past 24 hours" would have published.

## Write the live checks for Swagger Codegen and NSwag

Swagger Codegen writes test files you fill in. Its Python client came with
`test/test_default_api.py`, one method per operation, each with a body of `pass`, so the
scheduled job would be green from the first night:

```python title="swagger-codegen/test_default_api.py"
    def test_list_books(self):
        """Test case for list_books

        """
        pass
```

Fill each stub with a call against `configuration.host`, which the client defaults to the
document's literal `http://{sandbox}/v1`. Run it as `java -DapiTests=false -jar ...` if you would
rather not ship empty tests; the stub file is then left out.[^2]

NSwag writes no tests and, for this document, no code for the `X-API-Key` scheme. Its TypeScript
constructor takes a base URL and an object with a `fetch`, which is where the key goes:

```js title="smoke.mjs"
  const fetchWithKey = (url, init = {}) => fetch(url, { ...init, headers: { ...init.headers, 'X-API-Key': key } })
  const client = new Client(`http://${host}/v1`, { fetch: fetchWithKey })
```

`smoke.mjs` is the hand-written smoke list over that client. It tags what it creates with
`live-` and deletes it in a `finally`:

```bash
node smoke.mjs
```

```text output
list books: ok
create a tagged book: ok
load it back: ok
delete it: ok
records left: 0
```

## Check it worked

Rotate the sandbox key without updating the secret, start the live workflow by hand, then push a
release tag. The tests run that sequence against the stand-in. They also check the gate's edges,
the sweep, and the workflow shape:

```bash
node --test live.test.mjs | tail -8
```

```text output
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 21511.298524
```

`live.mjs` removes `NODE_TEST_CONTEXT` before it starts the inner `node --test`. With it
inherited, the inner run reports to the outer one and exits 0 on a red suite.

## When it goes wrong

The sandbox hits its record quota within a week. A delete failed inside the suite, and the
suite does not retry it. `cleanup.mjs` makes the sandbox answer one DELETE with 500 for each suite:

```bash
node cleanup.mjs
```

```text output
generated live suite: red, POST bodies [{}]
  left after the suite: bk1 tagged=true
  sweep deleted 1, left after the sweep: none
smoke checks: delete it: 500
  left after the suite: bk1 tagged=true
  sweep deleted 1, left after the sweep: none
```

The generated flow builds its create body from the request schema's required fields and examples.
Bookshelf's `Book` declares neither, so the POST carried `{}`. The update step writes the
`Mark01-` value into `author`. A failed update alone does no harm, because the suite still runs its
delete. A record whose update and delete both fail carries no mark, so give the suite a tenant of
its own, or have the sweep delete everything in it.

The gate blocks every release, and the newest live run is weeks old. In a public repository
GitHub disables a scheduled workflow after 60 days without repository activity, so no run is
newer than the day it stopped.
Re-enable the workflow; the gate failing closed is the intended behavior.

## When not to do this

Do not point a live suite at production. The generated flow creates, updates, and deletes, and a
failed delete leaves a record behind, as `cleanup.mjs` shows.

Do not gate on a run that tested other code. A nightly run checks main as it stood that night,
so a release cut from a later commit needs a manual run first.

Do not adopt sdkgen for its live suite alone. If your SDK comes from Swagger Codegen, filling its
stubs keeps one generator and one test layout; with NSwag, a dozen smoke checks over its client
cover the calls customers make. Run Schemathesis beside either when the risk is the server
drifting from the spec.

## Related how-tos

- [Layer unit, integration, and live tests for an SDK](/howto/layer-unit-integration-and-live-sdk-tests)

- [Keep test and live API keys from crossing environments](/howto/keep-test-and-live-keys-apart)

- [Choose a release automation tool for an SDK repository](/howto/choose-a-release-automation-tool-for-an-sdk)

- [Test an API release against the spec it replaces](/howto/test-a-release-against-the-previous-spec)

## Last verified

Verified 2026-11-20 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3,
NSwag 14.8.0 and Swagger Codegen 3.0.85. Every output block is what the command preceding it printed. NSwag and Swagger Codegen were run once on `api.json` and their output committed. The
two workflows were not run on GitHub; a test checks their shape. Schemathesis was not run.

[^1]: GitHub's [schedule event](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule)
notes that runs can be delayed during periods of high load, and names the start of every hour as
such a period. It adds that some queued jobs may be dropped if the load is high enough. Every
schedule written `0 * * * *` makes the hour a little busier.

[^2]: Swagger Codegen's [selective generation page](https://github.com/swagger-api/swagger-codegen/blob/master/docs/generation-selective.md)
describes `-DapiTests` and `-DmodelTests` as options that default to true. They are Java system
properties rather than generator flags, which is why they go before `-jar`. The generated class
names itself in its docstring as `DefaultApi unit test stubs`.