How-to › Test and mock integrations

How to build a CI matrix for an SDK across six languages#

Test TypeScript, Python, Go, PHP, Ruby and Lua SDKs on two or three runtimes each in one GitHub Actions matrix with one required status.

Audience
Platform team
Level
intermediate
Topic
Run SDK test suites
Languages
TypeScript, Python, Go, PHP, Ruby and Lua
Verified

A user reports that the Python SDK fails on Python 3.14. CI never ran 3.14, because each of the TypeScript, Python, Go, PHP, Ruby and Lua SDKs is tested on one runtime, in a job of its own. A red Lua job is one mark among many, so a branch rule has no single check to require.

What you get

You will end up with one workflow that tests the six SDKs on two or three runtimes each, reports one status, and keeps a failure inside its own cells. This is for you if your team runs CI for a repository that holds SDKs in several languages.

Short answer

Write the GitHub Actions matrix as include entries, one per language and runtime version, set fail-fast to false, install each toolchain with its own setup action, and cache each language on its manifest. Add one job that needs the matrix, runs if: always(), and fails unless every cell succeeded. Count the jobs before you push, then break one language on purpose and confirm only its cells go red.

You will need

A repository with each SDK in its own directory and a make test target in each, which is the layout sdkgen generates, and GitHub Actions or GitLab CI. The sample’s scripts need Node 22 or later and run offline. The workflows were checked by those scripts and not run on either service. The shell lines of the Composer cache step ran locally, against Composer 2.8.12.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
GitHub Actions matrixOne repository holds every SDK, and the code is on GitHubOne workflow with a setup step per language, each gated on matrix.langThe SDKs live in separate repositories with separate owners
GitLab CI parallel:matrixThe repository is on GitLab, with an image per runtimeOne job per language, and setup scripts inside images you trustYou need setup actions rather than container images
openapi-typescript and openapi-fetchTypeScript is the only SDK, generated as typesCovers one language, so the other five cells need generators of their ownSeveral languages must come from one tool
sdkgen generated CIThe six SDKs come from one sdkgen model in one repositoryA generated job per target, one runtime each, and no single statusYou need two or three runtimes per language today
Speakeasy target testingOne repository per language, each released on its own cadenceA licensed test feature, and one status per repository, six for six languagesOne status for all six languages, or no account

Voxgig maintains sdkgen. It is one of five options in the table, with its own limits stated in its row.

The single matrix keeps every language on one commit and one status, which suits a repository that releases together. A repository per language, as Speakeasy recommends, lets each language move on its own clock, at the price of six places to look. openapi-typescript is the cheapest cell to run and covers one of the six.

Read what sdkgen generates first

An sdkgen scaffold with the six targets writes .github/workflows/ci.yml, committed here as generated/sdkgen-ci.yml. check-workflow.mjs reads a workflow and reports what a multi-language pipeline needs from it.

node check-workflow.mjs generated/sdkgen-ci.yml
generated/sdkgen-ci.yml: 26 jobs
  versions: ts 1, py 1, go 1, php 1, rb 1, lua 1
  actions pinned to a commit: 50 of 50
  required status: none
  8 problem(s)
    26 job(s) run on a -latest label, which moves to a newer image without a change here
    ts: one runtime version only (24)
    py: one runtime version only (3.13)
    go: one runtime version only (1.24)
    php: one runtime version only (8.3)
    rb: one runtime version only (3.3)
    lua: one runtime version only (5.4)
    no job that needs every other job and fails unless all succeeded

Every action is pinned to a full commit SHA. The 26 jobs cover 26 targets, not only the six here, and each skips its steps through hashFiles when its directory is absent, so 20 of them start and check out for nothing. The rest of the list is the work.

Write the matrix as include entries

One include entry per cell names the language and the runtime. With no other axis, GitHub runs exactly the listed entries.

    strategy:
      # One language failing must not cancel the others.
      fail-fast: false
      matrix:
        include:
          - { lang: ts, version: '22' }
          - { lang: ts, version: '24' }
          - { lang: py, version: '3.12' }
          - { lang: py, version: '3.13' }
          - { lang: py, version: '3.14' }
          - { lang: go, version: '1.26' }
          - { lang: go, version: '1.27' }

The Node, Python, Go and Ruby versions come from the cached tools in the Ubuntu 24.04 runner image, committed under runner/. The image carries PHP 8.3 and no Lua, so setup-php and gh-actions-lua install the rest. Each language gets its setup action behind if: matrix.lang: actions/setup-node, setup-python, setup-go, shivammathur/setup-php, ruby/setup-ruby, and leafo/gh-actions-lua with leafo/gh-actions-luarocks. fail-fast defaults to true, and then GitHub cancels every running cell when one fails.

The status job is what a branch rule requires:

  sdk-ci:
    needs: test
    if: always()
    runs-on: ubuntu-24.04
    steps:
      - run: test "${{ needs.test.result }}" = success

if: always() makes it report after a failed cell, since a job whose needs failed is otherwise skipped. The needs context gives success, failure, cancelled or skipped.

node check-workflow.mjs sdk-ci.yml
sdk-ci.yml: 15 jobs
  versions: ts 2, py 3, go 2, php 3, rb 2, lua 2
  actions pinned to a commit: 11 of 11
  required status: sdk-ci
  0 problem(s)

Count the jobs before you push

matrix.mjs expands a matrix by the include rules as GitHub’s reference words them, and GitLab’s parallel:matrix by its own. Its counts are read from that documentation, since neither service ran these files. Two mistakes sit in pitfall/: every axis crossed, and an os axis added to the include list.

node matrix.mjs sdk-ci.yml pitfall/crossed.yml pitfall/os-axis.yml --gitlab gitlab-ci.yml
sdk-ci.yml: 14 jobs (GitHub allows 256 per matrix)
  ts   22 24
  py   3.12 3.13 3.14
  go   1.26 1.27
  php  8.3 8.4 8.5
  rb   3.4 4.0
  lua  5.3 5.4
pitfall/crossed.yml: 54 jobs (GitHub allows 256 per matrix)
  ts   9 jobs
  py   9 jobs
  go   9 jobs
  php  9 jobs
  rb   9 jobs
  lua  9 jobs
pitfall/os-axis.yml: 3 jobs (GitHub allows 256 per matrix)
  lua  5.4@ubuntu-24.04 5.4@macos-15 5.4@windows-2025
gitlab-ci.yml: 14 jobs (GitLab allows 200 per parallel:matrix)
  ts   22 24
  py   3.12 3.13 3.14
  go   1.26 1.27
  php  8.3 8.4 8.5
  rb   3.4 4.0
  lua  5.3 5.4

Crossing six languages, three version slots and three operating systems starts 54 jobs. The os axis is worse. By the reference’s rules, each include entry fits every original combination, so each one overwrites the last, and three Lua jobs remain.1 Add an operating system as include entries, for the cells that need it.

GitLab has no single matrix across jobs, so gitlab-ci.yml gives each language a job with its own parallel:matrix and image:

py:
  extends: .sdk-test
  image: python:${PYTHON}
  variables: { SDK: py, PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip" }
  parallel:
    matrix:
      - PYTHON: ['3.12', '3.13', '3.14']
  cache: { key: "pip-$PYTHON", paths: [.pip/] }

A failed job already leaves the others running in GitLab, and the pipeline’s status is the one status.

Break one language on purpose

The six generated suites ran here, one runtime each, through run-cells.mjs, which runs each cell’s command and keeps going after a failure, as fail-fast: false does. The second run put raise RuntimeError("broken on purpose") at the top of the Python SDK’s __init__.py.

node run-cells.mjs summary results/clean.json results/py-broken.json
results/clean.json: 6 cells ran, 0 failed
  pass  ts   node 22.23.5    exit 0   # tests 226; # pass 217; # fail 0
  pass  py   Python 3.13.18  exit 0   205 passed, 88 skipped
  pass  go   go1.24.7        exit 0   ok github.com/voxgig-sdk/bookshelf-sdk/go/test
  pass  php  PHP 8.3.6       exit 0   OK, but some tests were skipped!; Tests: 260, Assertions: 711, Skipped: 65.
  pass  rb   ruby 3.3.6      exit 0   262 runs, 147698 assertions, 0 failures, 0 errors, 70 skips
  pass  lua  Lua 5.4.6       exit 0   206 successes / 0 failures / 0 errors / 2 pending
results/py-broken.json: 6 cells ran, 1 failed
  pass  ts   node 22.23.5    exit 0   # tests 226; # pass 217; # fail 0
  FAIL  py   Python 3.13.18  exit 2   12 errors
  pass  go   go1.24.7        exit 0   ok github.com/voxgig-sdk/bookshelf-sdk/go/test
  pass  php  PHP 8.3.6       exit 0   OK, but some tests were skipped!; Tests: 260, Assertions: 711, Skipped: 65.
  pass  rb   ruby 3.3.6      exit 0   262 runs, 147698 assertions, 0 failures, 0 errors, 70 skips
  pass  lua  Lua 5.4.6       exit 0   206 successes / 0 failures / 0 errors / 2 pending

The break reached Python alone, because each SDK carries its own runtime code. On GitHub, push the same break on a branch and expect the three Python cells and sdk-ci to go red while the other eleven finish green. Revert it before merging.

Test the same SDKs with Speakeasy or openapi-typescript

Speakeasy was not run; it needs an account. Its unit is a target, one per SDK. Its monorepo guide recommends a single SDK per GitHub repository for most uses, and gives each SDK its own generate and release workflows even in a monorepo. A target in .speakeasy/workflow.yaml takes testing: enabled: true, and its workflow file reference says that runs target-specific testing such as go test or pytest after generation, if test generation is licensed. The reference names no setting for runtime versions, so each repository needs its own matrix and reports its own status. That suits teams that release each language separately better than one matrix does.

openapi-typescript generates types and openapi-fetch is the runtime, so it fills the TypeScript cells alone. Add its --check flag to those cells: it fails when the committed types no longer match the document, and writes nothing.

npx openapi-typescript api.json -o ts-types/schema.d.ts --check
✨ openapi-typescript 7.13.0
npx openapi-typescript api-changed.json -o ts-types/schema.d.ts --check
✨ openapi-typescript 7.13.0
 ✘  Generated types are not up-to-date!

The openapi-fetch page lists Node 18.0.0 or later, so both Node cells sit inside its support. Where TypeScript is the only SDK, this pair needs one runtime setup instead of six.

Check it worked

The tests hold the expander to GitHub’s own worked example, the workflow to its job count and its pins, and the recorded runs to a failure that stays in the Python cell. Two more run the Composer cache step under bash -e against a stand-in for Composer, and refuse the old form of that step.

node --test ci.test.mjs
1..11
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0

When it goes wrong

The Node cells fail at setup once setup-node is given cache: npm. That cache keys on a lockfile, and sdkgen’s generated TypeScript SDK ships no package-lock.json. Cache ~/.npm with actions/cache keyed on ts/package.json, as sdk-ci.yml does. The PHP target’s .gitignore drops composer.lock on purpose, so the Composer cache keys on composer.json too.

The Composer cache step reports success and writes an empty directory. composer config cache-files-dir reads ./composer.json, so at the repository root it fails. It was run locally at a root that held only php/composer.json. There it reported that the file cannot be found in the current directory, and exited 1. Wrapped as echo "dir=$(...)", the same line exited 0 and wrote dir=, because the line takes the status of echo. The cache step then gets an empty path. sdk-ci.yml passes --working-dir php, as its install step does, and assigns the result before writing it. GitHub runs a step with no shell key as bash -e {0}, by its workflow syntax reference, so a failed assignment stops the step. check-workflow.mjs refuses both shapes.

A cell’s toolchain changes under it with no commit to blame. The runner image readme committed under runner/, for image version 20261116.376.1, announces that the ubuntu-latest label will use Ubuntu 26.04 in November 2026. Pin ubuntu-24.04 and move deliberately.

The status check never reports. Without if: always(), the status job is skipped when a cell fails, and a branch rule that requires it waits. Keep always() and test needs.test.result.

When not to do this

Do not cross languages with operating systems. Six languages on three systems is a bill for cells nobody reads; add an operating system per language where its users need one.

Do not keep sdkgen’s generated workflow as your only CI if you promise more than one runtime per language. It tests one version of each, so the second runtime has to come from a matrix you add.

Do not force one matrix on SDKs that release on their own schedules. A repository per language, the shape Speakeasy’s targets take, gives each language its own status, on its own release clock.

Do not list a runtime nobody will fix. A cell is a promise to users, so Ruby 4.0 or Lua 5.3 in the matrix means somebody owns them when they go red.

Last verified

Verified 2026-11-21 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @voxgig/apidef 8.32.4, openapi-typescript 7.13.0, TypeScript 5.9.3 and yaml 2.9.1. Every output block is what the command preceding it printed. The six generated suites ran in a Linux sandbox twice, before and after Python was broken. They ran on Node 22.23.5, Python 3.13.18, Go 1.24.7, PHP 8.3.6, Ruby 3.3.6 and Lua 5.4.6, with busted and dkjson from Ubuntu’s apt packages. Their results are committed under results/, which the captured commands read without those runtimes. The Composer cache step’s lookup ran locally with Composer 2.8.12 on PHP 8.3.6, at a root holding only php/composer.json, with and without --working-dir php. Neither workflow was run on GitHub or GitLab. Speakeasy was not run, and how it tests a target is read from its documentation.

Footnotes

  1. GitHub’s workflow syntax reference adds an include entry to each combination it can join without overwriting an original value, and notes that added values can be overwritten. Its worked example, on the page about running job variations, uses fruit, animals, colors and shapes, and ends with {fruit: banana, animal: cat} as a combination of its own. It does not join {fruit: banana}, which arrived by include and so was never an original. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.