# How to ship a DataFrame-first Python client for your analysts > Generate a package where one call drains a paginated endpoint into a typed pandas frame, against a helper over `openapi-python-client` and a dlt source. Source: https://voxgig.com/howto/dataframe-first-python-client-from-openapi - Audience: api-producer - Level: intermediate - Languages: python - Verified: 2026-10-20 - Published: 2026-10-17 ## Short answer Add sdkgen's `py-data` target beside the `py` target, add the `paging` feature, and regenerate to get a Python package that returns typed pandas frames. Its `data()` function returns a client with one method per entity: `ad.applicants()` drains every page into a frame typed from the model. Dates stay strings until you pass `parse_dates`. A helper over `openapi-python-client` reaches the same frame in about thirty lines you own. --- ## You will need Node 22 or later for the generator, Python 3.12 or later, and the OpenAPI 3 document for your API. The document on this page, `api.json`, describes a recruiting API with `applicants` and `jobs` behind a cursor-paginated envelope of `data`, `next_cursor`, and `has_more`. Verified 2026-10-20 against Node 22.22.0, Python 3.13.16, @voxgig/sdkgen 4.35.0, @voxgig/create-sdkgen 0.30.8, pandas 3.0.6, `openapi-python-client` 0.29.1, httpx 0.28.1, and `attrs` 26.1.0. Both routes run against `mock_api.py`, a local stand-in for the API, so nothing here calls a network service. dlt is described from its documentation and was not run. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [dlt rest_api source](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api/basic) | Analysts want yesterday's rows too, loaded incrementally into a local DuckDB file | A second tool with its own state directory and schema inference, and a frame only after `pipeline.dataset()` | The analyst wants the live table in one call, with no file between the API and the frame | | [`openapi-python-client` plus a `to_dataframe` helper](https://github.com/openapi-generators/openapi-python-client#readme) | You already ship this client, or want no second generator | Pagination, unwrapping, and `dtypes` are code you write, test, and keep current with the schema | Several entities need frames, so the helper grows one branch per entity | | [sdkgen py-data target](https://voxgig.com/sdk) | Your SDK is generated by sdkgen and your analysts work in pandas | pandas only, a dependency on the sibling `py` package, and flattening rules the generator owns | Analysts work in Polars, or your API has no list operations to frame | Voxgig maintains sdkgen. This page compares it with `openapi-python-client` and dlt. The hand-written helper costs you the pagination loop and the `dtype` table, and every schema change lands on whoever maintains them. dlt puts a pipeline and a DuckDB file between the API and the notebook. That is the right shape for a nightly load and the wrong one for a cell that wants the current table. The py-data target costs a second generated package and a pandas-only output. Its flattening and typing rules come from the model rather than from you, which is the saving and also the loss of control. ## Add the data target beside the Python target The data package wraps the Python SDK rather than replacing it, so the two targets are generated from one model. `generate.sh` is the whole build, run once with network access; its output is committed under `generated/` so the rest of the page runs offline. ```bash title="generate.sh" npx --yes @voxgig/create-sdkgen@0.30.8 hirewire --def api.json --target py --target py-data (cd hirewire-sdk/.sdk && npm run add-feature paging && npm run generate) ``` Two flags name two targets, and the scaffold records them in the model, which is the other way to add one: list it beside `py` and regenerate. ```aontu title="generated/model/target-index.aontu" @"./py.aontu" @"./py-data.aontu" ``` The order of the two steps matters. The `paging` feature is not in a fresh model until you add it, and the frame methods drain whatever paging state that feature records. A model without it yields one page per call. The target also refuses to build alone: without `py` in the same model the generate step stops with `py-data requires the py target in the same SDK`. The entity model is where the column types come from. For `applicant`, `score` is `$INTEGER`,[^1] and `applied_at` is a `$STRING` whose format the model records as `fo: "date-time"`: ```aontu title="generated/model/applicant.aontu" applied_at: { fo: "date-time" h: "Applied At" n: "applied_at" r: true t: "`$STRING`" } ``` The data target maps `$INTEGER` to pandas `Int64`, `$BOOLEAN` to `boolean`, and every `$STRING` to `string`, format included, so date parsing is left to the caller. The generated `hirewire_data` package holds the methods, a `README.md` with a column table per entity, an `AGENTS.md`, and `notebooks/quickstart.ipynb`, all committed here. ## Pull every page into one frame `frames_demo.py` builds the client the way a notebook would, with the credential made explicit and the base URL pointed at the stand-in. The paging feature is switched on through the options the SDK constructor takes. ```py title="frames_demo.py" ad = data(token="test-key", base_url=mock_api.base_url(api), feature={"paging": {"active": True, "limit": 2}}) df = ad.applicants(quiet=True) ``` ```bash PYTHONPATH=vendor:generated/py:generated/py-data python3 frames_demo.py ``` ```text output rows: 5, from 3 requests GET /v1/applicants?limit=2&page=1 GET /v1/applicants?cursor=c2&limit=2 GET /v1/applicants?cursor=c4&limit=2 id stage score remote applied_at location.city 0 app_01 interview 82 True 2026-03-02T07:30:00Z Lagos 1 app_02 screening False 2026-03-02T08:00:00+02:00 Utrecht 2 app_03 applied 67 True 2026-03-01T22:15:00Z Taipei 3 app_04 offer 91 False 2026-02-27T16:45:00Z Haifa 4 app_05 rejected True 2026-02-26T09:05:00Z Nairobi dtypes applied_at string email string id string job_id string name string remote boolean score Int64 stage string location.city str location.country str applied_at with parse_dates: datetime64[us, UTC] ``` Five rows arrived in three requests of two. The first request carries `page=1`, because the paging feature assumes page numbers until a response hands it a cursor; the API ignores the parameter and answers the first page. Each later request carries the `next_cursor` the previous response returned, until `has_more` is false. Read the `dtypes` block against the schema. `score` is `Int64`,[^2] so the two nulls print as `` and the column stays integral. `remote` is `boolean`, and every declared string is `string`. The nested `location` object was flattened one level into `location.city` and `location.country`, the two columns pandas typed on its own, since the model declares `location` as one object field. `applied_at` is a string column until the last line, where `parse_dates` turns it into a UTC `datetime`. ## Write the same frame by hand over the generated client The comparison client was generated from the same `api.json` into `generated/opc`. It gives you a `Client`, one module per operation, and `attrs` models with `to_dict()`. The frame is yours. ```py title="to_dataframe.py" def applicants_frame(client: Client, page_size: int = 2) -> pd.DataFrame: """Every applicant as one row. The loop, the unwrapping and the typing are yours.""" rows: list[dict] = [] cursor = UNSET seen: set[str] = set() while True: page = list_applicants.sync(client=client, cursor=cursor, limit=page_size) if page is None: raise RuntimeError("list_applicants answered with an undocumented status") # to_dict() drops fields the API did not send, so a column can be missing # from one page and present in the next. rows.extend(applicant.to_dict() for applicant in page.data) if not page.has_more or page.next_cursor in (None, UNSET): break if page.next_cursor in seen: raise RuntimeError(f"pagination repeated cursor {page.next_cursor!r}") seen.add(page.next_cursor) cursor = page.next_cursor return pd.json_normalize(rows, sep=".") def typed(df: pd.DataFrame) -> pd.DataFrame: """The dtype work py-data reads off the model, done here by hand and by name.""" out = df.convert_dtypes() # Without format="ISO8601" pandas takes the format from the first value and # raises on any value that carries fractional seconds when that one did not. out["applied_at"] = pd.to_datetime(out["applied_at"], utc=True, format="ISO8601") return out ``` ```bash PYTHONPATH=vendor:generated/opc python3 to_dataframe.py ``` ```text output rows: 5, from 3 requests json_normalize after typed() id str string job_id str string name str string stage str string applied_at str datetime64[us, UTC] email str string score float64 Int64 remote bool boolean location.city str string location.country str string applied_at as to_dict() wrote it: 2026-03-02T07:30:00+00:00 ``` The left column is what [`json_normalize`](https://pandas.pydata.org/docs/reference/api/pandas.json_normalize.html) infers on its own: `score` has become `float64` because one page carried a null, and the dates are text. The right column is after [`convert_dtypes`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.convert_dtypes.html) and one [`to_datetime`](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html) call, which matches the generated method with `parse_dates` on every declared column. The difference is who names the columns: `typed()` knows `applied_at` by hand, and a renamed field in the schema breaks it without a test to say so. The last line shows the models round-tripping the timestamp, so `Z` on the wire comes back as `+00:00`.[^3] The helper also stops at a repeated cursor, which would otherwise keep the cell running forever. ## Check it worked `test_frames.py` runs both routes against the mock API. The test that guards the whole point compares the frame's `dtypes` with a table built from the OpenAPI schema. A second one asserts that a filter matching nothing still returns the declared columns with the declared `dtypes`. ```py title="test_frames.py" def test_declared_columns_carry_the_schema_dtypes(self): df = self.ad.applicants(quiet=True) self.assertEqual({c: str(df[c].dtype) for c in SCHEMA_DTYPES}, SCHEMA_DTYPES) ``` ```bash PYTHONPATH=vendor:generated/py:generated/py-data:generated/opc python3 test_frames.py ``` ```text output test_the_helper_lands_on_the_same_rows_and_dtypes (__main__.HandWrittenHelper.test_the_helper_lands_on_the_same_rows_and_dtypes) ... ok test_the_helper_parses_timestamps_with_and_without_fractions (__main__.HandWrittenHelper.test_the_helper_parses_timestamps_with_and_without_fractions) ... ok test_the_helper_refuses_a_cursor_that_repeats (__main__.HandWrittenHelper.test_the_helper_refuses_a_cursor_that_repeats) ... ok test_a_filter_that_matches_nothing_keeps_the_declared_columns (__main__.PyDataFrames.test_a_filter_that_matches_nothing_keeps_the_declared_columns) ... ok test_a_null_integer_stays_an_integer_column (__main__.PyDataFrames.test_a_null_integer_stays_an_integer_column) ... ok test_dates_are_strings_until_parse_dates (__main__.PyDataFrames.test_dates_are_strings_until_parse_dates) ... ok test_declared_columns_carry_the_schema_dtypes (__main__.PyDataFrames.test_declared_columns_carry_the_schema_dtypes) ... ok test_every_page_is_drained_into_one_frame (__main__.PyDataFrames.test_every_page_is_drained_into_one_frame) ... ok test_one_level_of_nesting_becomes_dotted_columns (__main__.PyDataFrames.test_one_level_of_nesting_becomes_dotted_columns) ... ok ``` Nine tests print `ok` and the runner ends with `OK`. The paging test also counts requests at the mock, because three requests for five rows is what proves the drain. ## When it goes wrong A date column sorts wrongly and `.dt` raises. The method typed `applied_at` as `string`, because the data target never infers dates from a string format. `pitfall.py` shows the cost with two applicants who applied the same morning in different offsets. ```bash PYTHONPATH=vendor:generated/py:generated/py-data python3 pitfall.py ``` ```text output 1. applied_at stays a string column until you ask for dates dtype: string latest by string order: Bram de Vries 2026-03-02T08:00:00+02:00 latest by parsed order: Ada Okafor 2026-03-02 07:30:00+00:00 .dt on the string column: Can only use .dt accessor with datetimelike values parse_dates when one value has milliseconds: ['2026-03-02 07:30:00+00:00', 'NaT'] 2. without the paging feature, one page comes back and nothing complains rows: 2, from 1 request the API saw: ['/v1/applicants'] ``` Sorted as text, the `+02:00` timestamp lands after the `Z` one, so string order names the second applicant as the most recent when the first applied ninety minutes later. Pass `parse_dates=["applied_at"]` only when every value has one precision, since it turns any other format into `NaT`, as the first block's last line shows. Otherwise call `to_datetime` with `utc=True` and `format="ISO8601"`, as `typed()` does. The frame has exactly one page of rows. The client was built without the `paging` feature, so the drain made one call and returned it, which the package treats as an API with one page. Pass `feature={"paging": {"active": True}}` to `data()`, and check that the model carries the feature at all. The generate step fails on `py-data` with a message about the `py` target. The data package imports the SDK that target writes, so add it with `npm run add-target py` and regenerate. ## When not to do this Do not ship sdkgen's data target to analysts who work in Polars. pandas is the only frame library the generated package knows, declared as a hard dependency with the reason written into the target model. Those analysts are better served by the `openapi-python-client` helper with [`polars.from_dicts`](https://docs.pola.rs/api/python/stable/reference/api/polars.from_dicts.html) in place of `json_normalize`. Do not generate a frame package for an API whose entities have no `list` operation. The methods come from `list` and `load`; an API of actions gets an empty class and an `ad.frame()` escape hatch. Do not read the generated `dtypes` as validation. A column that will not convert to its declared type is left as the API sent it, by design, so a frame can look typed and carry one object column. The `dtype` test on this page is how you notice. Do not adopt a second generated package to save one helper for one entity. The helper here is thirty lines; the package is a release, a PyPI distribution, and a sibling dependency to keep in step with the SDK. The target pays off when several entities need frames and the API keeps changing. Do not reach for dlt to fill a notebook cell. Its [`rest_api` source](https://dlthub.com/docs/dlt-ecosystem/verified-sources/rest_api/basic) and [`pipeline.dataset()`](https://dlthub.com/docs/general-usage/dataset-access/dataset) are built for loads that run on a schedule and keep history, and the frame arrives from the DuckDB file rather than from the API. ## Related how-tos - [Describe pagination so generators can follow it](/howto/describe-pagination-in-openapi-for-generators) - [Loop over every page of a REST collection](/howto/loop-over-every-page-of-a-collection) - [Generate Python docstrings and type hints from an API spec](/howto/python-docstrings-and-type-hints-from-a-spec) - [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen) ## Last verified Verified 2026-10-20 against Node 22.22.0, Python 3.13.16, @voxgig/sdkgen 4.35.0, @voxgig/create-sdkgen 0.30.8, pandas 3.0.6, `numpy` 2.5.4, requests 2.34.2, `openapi-python-client` 0.29.1, httpx 0.28.1, and `attrs` 26.1.0. Every output block is what the command preceding it printed. The scaffold and both generators ran once, in `generate.sh`, and their output is committed under `generated/`; the page's commands read those files and run against the local stand-in API. The generated quickstart notebook was read and not executed, and dlt is described from its documentation. [^1]: The sample document is written to [OpenAPI 3.0.3](https://spec.openapis.org/oas/v3.0.3), whose Schema Object has a `nullable` field. A true value adds null to the allowed type, only if `type` is explicitly defined within the same Schema Object, and the default is false. [OpenAPI 3.1.0](https://spec.openapis.org/oas/v3.1.0) made the Schema Object a superset of JSON Schema Draft 2020-12 and lists no such field, so the same column is written there as a type array. The generator read the 3.0.3 form as `$INTEGER`, and the frame holds the null either way. [^2]: The nullable integer type the frames rest on is spelled `Int64`. The [pandas documentation](https://pandas.pydata.org/docs/user_guide/integer_na.html) asks the reader to note the capital letter, which is all that separates it from NumPy's `int64` in print. The same page explains the need: a missing value forces an integer array to become floating point, and some integers cannot be represented as floating point numbers at all. A note beneath marks `IntegerArray` experimental, with an API or implementation that may change without warning. [^3]: [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.1) spends its section 5.1 on ordering, and makes a promise with conditions. Date and time strings sort into time order as strings only under three conditions: one time zone, one spelling of it, all `Z` or all `+00:00`, and one count of fractional digits. The mock API breaks the first condition on purpose, with one applicant at `+02:00`, and the pitfall run shows what string order does with it. Section 4.3 adds that `Z` and `+00:00` mean the same thing, so the round trip through the generated models changed the spelling and nothing else.