Skip to content

Emit Python JSON-Schema models as dataclasses with transfer type converters - #125

Open
bergundy wants to merge 17 commits into
ts-transfer-type-encodingfrom
python-dataclasses
Open

Emit Python JSON-Schema models as dataclasses with transfer type converters#125
bergundy wants to merge 17 commits into
ts-transfer-type-encodingfrom
python-dataclasses

Conversation

@bergundy

@bergundy bergundy commented Aug 15, 2026

Copy link
Copy Markdown
Member

Stacked on #124 (ts-transfer-type-encoding) — the base is that branch, not main. Merge after #124.

Summary

Generated Python JSON Schema models now use slotted, keyword-only dataclasses instead of pydantic.BaseModel. Their wire contract lives in generated transfer type converters discovered by Temporal's default data converter, so consumers no longer need Pydantic or a custom Pydantic data converter.

@_transfer_type_convertible(_UserTransferTypeConverter)
@dataclasses.dataclass(slots=True, kw_only=True)
class User:
    name: str
    location: Location | None = None

The generated API now follows these rules:

  • Optional or nullable values are annotated as T | None.
  • Schema defaults are materialized through mutable properties backed by private presence fields. Reading an unset property returns its schema default, assigning stores an explicit value, and del model.field restores the unset state.
  • Default constants are no longer part of the generated public API.
  • Unknown object members are preserved in additional_properties.
  • Validation failures aggregate into the shared ValidationError / Violation { path, reason } shape.
  • ValidationError and Violation are re-exported from every generated package root, so consumers do not import the private _definitions module.
  • The default Temporal converter performs the generated model-to-wire and wire-to-model conversion.

This keeps materialized reads separate from wire presence: an unset default-bearing property reads as its schema default but remains omitted when serialized, while an explicitly assigned value is emitted.

Correctness fixes found during the rewrite

The dataclass conversion exposed several existing cross-language and wire-compatibility gaps. This branch also:

  • validates temporal values consistently on parse and serialization, including Python 3.10 sub-second behavior, year bounds, oversized duration components, naive datetimes, and invalid timedeltas;
  • rejects non-finite JSON numbers and normalizes integral closed numeric values;
  • validates array elements inside unions, rejects values matching no union branch, validates before serializer dispatch, and preserves nested serialize-side error aggregation;
  • prevents authored property names from shadowing converter locals, builtins, imports, or synthesized module names;
  • uses tuple membership for const/enum checks and keeps generated Python Ruff-clean;
  • enforces a TypeScript string array element's own constraints;
  • compares canonical wire bytes in the Python fixture suite instead of relying on parsed-value equality; and
  • runs the Python samples on the declared Python 3.10 floor and adds ruff check to the repository validation script.

Every generated-output change was made in the generator and regenerated with cargo build-json-examples --lang python.

Review

Best reviewed one commit at a time, starting at 93f55d8. The final generated-default API is concentrated in:

  • 4a2ab03 — materialize schema defaults through properties;
  • 988fc7e — retain T | None union syntax;
  • 4cbd8f9 — clear explicit values with property deleters;
  • e439348 — compact the Unreleased changelog entry; and
  • 559eb71 — export validation error types from generated package roots.

Verification

./scripts/validate.sh passes under the nvm-managed Node 26 runtime. This includes:

  • Rust formatting and the full cargo test --features advanced suite;
  • Python tests on 3.13 and the declared 3.10 floor;
  • BasedPyright with zero errors and warnings;
  • Ruff check and format verification;
  • TypeScript typechecking, Vitest, and Prettier; and
  • Go formatting and tests for both sample tiers.

Known follow-up

src/generator/proto/python.rs still emits the bare transfer_type_convertible decorator for WIT-generated Python. Its circular static typing issue is separate from this JSON Schema rewrite and remains follow-up work.

@bergundy
bergundy requested a review from a team as a code owner August 15, 2026 04:16
@bergundy
bergundy marked this pull request as draft August 16, 2026 21:49
@bergundy
bergundy marked this pull request as ready for review August 17, 2026 22:07
@bergundy
bergundy requested a review from tconley1428 August 18, 2026 15:49
@bergundy
bergundy force-pushed the python-dataclasses branch from 559eb71 to 3244444 Compare August 19, 2026 21:19
…erters

Generated Python models were the only emitter output requiring a contrib
dependency: every model was a strict `pydantic.BaseModel` carrying its wire
contract as `ConfigDict`, `Field(alias=...)`, `Annotated` validator aliases,
five flavors of `model_validator`, and a `model_serializer` keyed on
`model_fields_set`. That forced `pydantic` on consumers, forced the non-default
`pydantic_data_converter`, and left Python the odd one out on error aggregation.

Models are now plain `@dataclasses.dataclass(slots=True, kw_only=True)` types
whose entire wire contract lives in a generated
`_<Model>TransferTypeConverter`, registered on the class so the *default*
Temporal data converter picks it up with no user wiring. This is structurally
the TypeScript design -- inert data plus an off-model converter owning both
directions -- expressed through the SDK's transfer-type hook. Validation
aggregates into a generated `ValidationError` over `Violation { path, reason }`,
so all four targets now share one error shape.

Two deliberate behavior changes, both keeping Python in line with the other
targets rather than ahead of them:

- Optional+nullable collapses. A dataclass has no presence channel, so absent
  and explicit `null` both read as `None` and both re-serialize as omitted.
  This extends P1's exception (a) from Go/Java to Python.
- `default` is advisory, as in TypeScript. The field stays `T | None = None`
  and the value moves to a module-level `DEFAULT_<FIELD>` constant. Baking the
  default into the field would have made Python's wire a superset of the shared
  fixtures -- a third exception to round-trip byte-identity, where P1 allows
  two. Python trades pydantic's free materialize-on-read for byte-identity.

`additionalProperties` becomes an explicit `additional_properties` member,
matching Go/Java/TS and folding Python into the catch-all name collision that
the loader already rejected for it.

The runtime `_definitions.py` wraps `transfer_type_convertible` in a shim that
erases the converter's value-type parameter: binding it directly on the
decorated class is circular for a static type checker and degrades every model
to `Unknown`. `src/generator/proto/python.rs` has the same latent issue, benign
only because a test imports those models first.

Also corrects pre-existing spec drift found while auditing the Python cells:
the string-length reason was documented as `length must be <= N, got M` for all
four languages where every emitter emits `must have length <= N, got M`;
`required.md` documented a `required property "x" is missing` reason that no
emitter produces; `additionalProperties.md` claimed Python was exempt from the
catch-all collision reject that the loader applies to every language; and
`uniqueItems.md` described a hash-set membership test that raises on the
unhashable dataclass elements Python now generates.
The generated temporal helpers validated with the pinned regex and then handed
the string to a `datetime` parser that could still raise, so three classes of
input escaped the converter as a bare `ValueError` instead of the aggregated
`ValidationError` (P11) -- and the serialize direction had no validation at all,
so an unchecked dataclass silently produced wire bytes its own parser rejects
(P12).

- Year 0000. `datetime.MINYEAR` is 1, so the value has no Python
  representation; `_valid_temporal_calendar` now rejects it and
  `_temporal_reason` names the limit. Go/TypeScript/Java all accept year 0, so
  this is a genuine per-language accept-set divergence Python cannot avoid.
- Sub-second precision. Before 3.11 `fromisoformat` parses only the fraction
  widths `isoformat` writes, so RFC 3339 `.1` and `.1234567` raised on the
  declared 3.10 floor while every other target accepted them.
  `_temporal_isoformat` pads or truncates the fraction to 6 digits (and folds in
  the `Z` rewrite), so the accepted set no longer depends on the interpreter
  version. Digits past the sixth are dropped at `datetime`'s own resolution --
  the loss P1 exception (b) allows, mirroring Go's truncation at nanoseconds.
  Canonical output is unchanged: `_temporal_frac` re-trims, so `.1` still writes
  as `.1`.
- Duration overflow. CPython refuses `int()` on more than 4300 digits, so the
  guard now bounds the magnitude by digit count before converting (leading zeros
  stripped first, matching TypeScript's `Number()`), and a 5000-digit component
  produces a violation instead of crashing.
- Serialize-side representability. `_check_date_time`/`_check_time`/
  `_check_duration` hold a materialized value to what the narrowed grammar can
  spell -- a naive datetime, a sub-minute UTC offset, a negative, sub-second or
  over-cap duration -- and append a `Violation` under the field's own path
  rather than emitting bad data. `date` needs no predicate: every
  `datetime.date` writes a valid wire date.

The predicates land in the shared `Validate` layer (P12.2) via
`render_py_field_checks`, so the call site stays a single expression and the
serialize path picks them up for declared properties, nullable members and typed
map members alike, with the correct path and full aggregation.
Two wire-compatibility fixes in the JSON-Schema Python emitter.

`multipleOf` on a `number` crashed instead of validating. Python's
`json.loads` accepts the `Infinity`/`-Infinity`/`NaN` literals its dialect
adds, and `math.fmod(inf, n)` raises `ValueError` (an integer literal past
the binary64 range raises `OverflowError`), so untrusted wire bytes escaped
the aggregated `ValidationError` (P11). A `number` without `multipleOf` was
worse: `inf`/`nan` parsed and re-serialized verbatim, bytes that Go's
`json.Unmarshal`, `JSON.parse` and Jackson all reject (P1). The shared
numeric checks now guard finiteness first and hang the remaining predicates
off its `else`, so the check runs in both directions (P12). This converges on
Go, which rejects every non-finite and out-of-binary64-range `number` on
parse and refuses to marshal one on the way out.

An integer `const`/`enum` kept the wire float: `{"revision": 1.0}` stored
`1.0` and re-serialized as `1.0`, where Go (`parseIntegerField`) and Java
(`SpecNumbers.specLong`) normalize to an integer and emit `1`. A closed
numeric set now routes the wire value through `_parse_spec_integer` before
the membership comparison, which also reinstates the `1.5` reject and the
integer cap those fields bypassed. Float-valued sets have no `Literal` form
(PEP 586) and keep the wire value as it arrived.

The checked-in samples are regenerated separately.
… alias docs

Four fixes to the generated Python JSON-Schema union layer and serialize path.

1. A union's array branch now decodes elementwise (P1). It cast the whole
   value to `list[float]` and ran only `minItems`/`uniqueItems`, so any list
   was admitted: `{"measurements": ["a", "b"]}` round-tripped verbatim while
   Go decodes `[]float64` and Java binds a typed list, both rejecting. The
   branch reuses the same element parse a declared array member runs (split
   out of `render_array_parser` as `render_py_array_elements`, which the token
   guard lets the union call without re-testing `isinstance(value, list)`), so
   a bad element reports at its own index (`measurements[0]`). This also
   resolves the `True == 1` uniqueness discrepancy at its root: a `bool` in a
   `list[float]` is now rejected as a bad element before `_check_unique_items`
   sees it.

2. Serialize-side nested violations are re-pathed and aggregated (P11/P12).
   `to_transfer_type` wrapped no nested conversion, so a nested
   `ValidationError` propagated raw -- discarding the parent's already
   collected violations and its own path prefix. Every nested conversion --
   `$ref` members, array elements, typed-map members and union dispatchers --
   now runs under a `try` that funnels through `_collect`, the analogue of
   Go's `mergeNested`: a `Showcase` with a bad `name` and a bad
   `location.city` reports both, under those paths, and a bad element reports
   at `segments[1]` / `rows[1].cell`.

3. A union's serialize now rejects a value matching no branch (P12). The
   dispatch still falls through to its last branch unguarded (that decision
   is deliberate and documented), but a value in *no* branch was emitted
   verbatim -- bytes every parser rejects. The terminal test widens the value
   to `object` first, so the guards are neither provably exhaustive nor
   redundant and basedpyright stays at zero errors and zero warnings.

4. A union `TypeAlias`'s docstring follows its assignment, as a module-level
   variable docstring must and as dataclass members already do. Emitted
   before it, each union's description documented the preceding statement and
   the last alias's text was dropped.

An optional+nullable member no longer repeats the caller's `is not None`
guard around its serialize-side checks, which pyright reports as an
unnecessary comparison.

Checked-in samples are intentionally not regenerated here.
A declared property named after one of the converter's own locals silently
disabled validation. `violations: list[Violation] = []` was rebound by the
`violations` property's slot (`violations: str | None = None`), so the collected
violations were thrown away and an invalid payload was returned as a model;
`raw`, `len`, `int`, `str`, `bool`, `dict`, `isinstance`, `typing`, `math` and
`out` crashed every payload instead.

Rather than blocklist names — a property may be named anything — the parse body
now holds each property's value in a `<member>_value` slot local, extending the
discipline the emitter already applied to its temporaries (`{slot}_raw`,
`{slot}_parsed`, …). No fixed local, builtin, imported module, or synthesized
module-level name ends in `_value`, and no derived temporary does either, so the
slot cannot collide with anything: the shadow is structurally impossible instead
of merely unlisted.

The module-level names the emitter synthesizes beyond `DEFAULT_<FIELD>` now
participate in the P15 collision pass, computed through the emitter's own naming
helpers so the check cannot drift from what is emitted:

- `_<MODEL>_DECLARED` — `to_shouty_snake_case` is not injective over verbatim
  `x-py-name` overrides, so `ContactPy` and `ContactPY` shared one frozenset and
  the loser's declared properties leaked into its catch-all (a P13 break).
- `_<base>_{from,to}_transfer_type` — likewise non-injective, and a named union's
  base can coincide with an inline `<model>_<member>` one. An inline union is now
  named from the member's *emitted* identifier, so an override moves it (P15's
  escape hatch has to reach every synthesized name).
- `_<Model>TransferTypeConverter` and `_PATTERN_<HEX>` — injective in their own
  family, but a verbatim override can spell either; the pattern constants are
  keyed by pattern text, so identical patterns still share one constant.
- the converter bodies' own locals, for the mirror-image case: a type overridden
  to `raw` is shadowed inside every body that parses one.

Also reserves `_definitions` as an input-module name alongside `definitions`:
Python emits its shared runtime as `_definitions.py`, so an input named
`_definitions.yaml` emitted a `_definitions/` package directory at that module's
own import path, shadowing it and breaking every generated
`from .._definitions import ...`.

The checked-in Python JSON samples are left unregenerated; the two snapshot
tests fail until they are rebuilt.
Applies the five review fixes to the checked-in generated output, and records
that both `definitions` and `_definitions` are reserved module names now that
the loader rejects either spelling for every target.

The regenerated diff is dominated by two mechanical changes -- the converter's
per-property local is now `<member>_value` rather than the bare member
identifier, and the serialize path assigns through statements so a nested
conversion can be wrapped -- plus the new finiteness, temporal and closed-value
checks.
The Python round-trip suite compared `json.loads(payload.data)` on both sides, so
every assertion ran on a parsed value and nothing verified the wire bytes. That is
the whole justification for one of the design decisions -- a schema `default` stays
off the dataclass field so Python's output is byte-identical to the shared fixtures
-- and it let a real bug through: an integer `const`/`enum` kept its wire `float`,
so `{"revision": 1.0}` re-emitted as `1.0`, and `1 == 1.0` in Python.

`json_converter_helper` now offers `encode_bytes` (the payload's own `data`) against
`canonical_fixture_bytes` (the fixture through `json.dumps(..., sort_keys=True,
separators=(",", ":"))`). Canonicalizing normalizes only insignificant whitespace
and member order -- neither part of the contract -- while preserving the numeric
form, string escaping and the presence of every key. Every per-suite round-trip
assertion moves to that comparison.

The two deviations are one central, documented table rather than a weakened
assertion: `COLLAPSED_NULL_MEMBERS` names the five fixtures carrying an explicit
`null` on an optional+nullable member (P1 exception (a)), and
`NON_CANONICAL_FIXTURES` the one fixture that exists to be normalized. A schema
`default` needs no entry -- an unset defaulted key is omitted, which is *why* the
bytes match. `test_wire_fixtures.py` sweeps all 48 fixtures, fails if one is added
without a model, and asserts each exception entry is both necessary and sufficient.

Per fix, a test that fails without it:

- Integer closed values normalize (`revision`/`tier` from `1.0`/`2.0` re-emit as
  `1`/`2`, asserted on bytes -- the parsed comparison passes either way), plus the
  `1.5` and integer-cap paths the closed set bypassed.
- Non-finite numbers are rejected in both directions, reached through raw wire text
  since `json.loads` accepts `Infinity`/`-Infinity`/`NaN`, plus `1e400` and a
  401-digit integer literal.
- Year 0000 is a violation naming `datetime.MINYEAR`; sub-second widths `.1`, `.12`
  and `.1234567890` are accepted and re-emit canonically; a 5000-digit duration
  component is a violation.
- Serialize-side temporal representability: a naive `datetime`, a negative,
  fractional and over-cap `timedelta`, and a sub-minute offset, each under the
  field's own path and aggregating.
- Serialize-side nested aggregation: one model with a flat failure plus
  `location.city`, `segments[1]`, `rows[1].cell` and `choices.a.kind` reports all
  five, fully pathed.
- A union's array branch types every element; a value in no branch is rejected on
  serialize rather than emitted.

The identifier-shadowing fix needs a schema no sample declares, so it lands in
`tests/generate_python.rs`: sixteen properties named after the converter's own
locals, builtins, imported modules and method parameters. The rendered output is
asserted for the `_value` slot mechanism, and the generated package is then *run*,
because the failure was silent -- a `violations` property rebound the accumulator
and an invalid payload came back as a model.

Finally the sample suite now also runs on 3.10, the declared floor. The existing
AST check validates syntax only and the project environment is 3.13, so nothing
would have caught the sub-second bug -- it raised on 3.10 alone. `uv run --python
3.10 --locked pytest` reuses the same lockfile in an environment under `target/`,
adding ~3s and no second lockfile to maintain.
…eason

Two lint/consistency fixes in the generated Python JSON-Schema layer, plus the
gate that let the first one through.

1. A `const`/`enum` check now tests membership in a tuple of the admissible
   values in both directions. The parse side chained one `!=` per member, so a
   boolean `const` emitted `enabled_value_raw != True` -- a lint error in the
   user's repository (ruff E712) and nothing like hand-written Python (P2) -- and
   a multi-member `enum` emitted one comparison per member. The `!=` chain would
   likewise emit `!= None` (E711) for any future null-valued set. Parse and
   serialize now share one shape (`py_value_tuple`), which also drops the magic
   trailing comma that exploded every multi-member serialize-side test across
   five lines. A `contains` matcher's `const` takes the same shape, where a
   boolean matcher would have emitted `element == True`.

2. A mistyped array element now reports the type it failed to be
   (`tags[0]`, `expected string`), as an element of every other type already did
   and as Java's element loop reports. A plain `string` element took a special
   case that reported a bare `expected element`, naming neither the expected type
   nor anything the element's own indexed path did not already carry; the special
   case existed only to mirror the TypeScript emitter, whose `expected element`
   branch already diverges from Python for a *constrained* string element (and
   drops that element's constraints entirely -- a separate defect, TypeScript
   being out of scope here). Every element kind now takes the same parse the
   value in that position takes anywhere else, and `items.md` states the reason
   convention.

`scripts/validate.sh` ran `ruff format --check` but never `ruff check`, so lint
defects in generated Python were ungated -- which is how the E712 survived. Both
Python tiers now run it. It also surfaced eight pre-existing E402s in the
advanced tier's hand-written tests, whose `wit.*` imports sat below two path
constants for no reason; they move up with the other imports, matching the
sibling test module.
A `string` element schema took a special case in the array parser: a bare
`typeof element !== 'string'` check followed by an assignment, skipping
`render_ts_string_checks` entirely. An
`items: { type: string, minLength: 3, pattern: "^[a-z]+$" }` array therefore
accepted `["a"]` and `["A"]` in TypeScript while Go, Python and Java rejected
both -- an accept-set divergence, which P1 makes part of the wire contract
rather than a per-language detail. The element's compiled `PATTERN_<HEX>`
constant was emitted and never referenced, dead code in the user's repository
and the independent signal that the constraint had been dropped.

Deleting the special case routes a `string` element through
`render_value_parser_at_depth`, the same parse every other element kind (and
every non-element position) already took. That applies the missing
`minLength`/`maxLength`/`pattern`/`format` checks at the element's own index,
and reports a mistyped element as `expected string` rather than the bare
`expected element`, which named neither the expected type nor anything the
indexed path did not already carry. Python removed the identical inherited
special case in e6948ca, so the two languages' element reasons are byte-equal
again -- the standing goal recorded in PRINCIPLES Python §2.

The regenerated samples change only that reason text: no checked-in schema has
a *constrained* string element, which is why the dropped constraints never
surfaced in a sample diff. The showcase suite gains the element-reason
assertion Python's suite already carries.
The rebase put two independent reworks of the same files together, and three
seams needed closing:

- `DEFAULT_<FIELD>` is now named off the **emitted member identifier** in
  Python as it already is in TypeScript, so an `x-py-name` override moves the
  constant with its member (P15) and the loader's shared replication of the
  rule matches what each generator emits. The loader also derives the
  identifier per language rather than always through TypeScript's mapping.
- The collision test that used two *models* no longer describes a collision:
  `DEFAULT_<MODEL>_<FIELD>` qualification separates them. It now pins the case
  qualification cannot resolve — two members of one model — in both languages,
  plus the override that opens the escape hatch, and keeps the two-model shape
  as an accepted case.
- A TypeScript-only converter-collision test asserted Python accepts the same
  schema. Python now derives module-level names from the type name too, and
  `HTTPError`/`HttpError` fold together in `_HTTP_ERROR_DECLARED`, so it
  rejects for its own reason.

Also updates two assertions the dataclass rewrite invalidated (`class Page(`
is now `class Page:`) and one showcase test still calling `ShowcaseMapper`.
A union serialize function guards every branch but the last and falls
through to it. For a *named* union that was already safe -- the function
ran the union's no-branch-matched test first, so the fallthrough was only
reached by a value some branch admitted.

An inline property-position union had no such test. The declaring
converter appended the `expected one of` violation and then handed the
same value to the dispatch, so a member holding a value no branch admits
reached the last branch's converter and raised whatever its first
attribute access raised -- `AttributeError: 'int' object has no attribute
'kind'`, which is not the `ValidationError` the caller catches, so it
propagated alone and discarded every violation already collected (P11).

Run the checks inside the serialize function for every union, named or
inline, ahead of the dispatch: that is the only place that can stop it.
The declaring member drops its now-duplicate check and keeps only the
`_collect` re-path, which reports the union function's empty-path
violations under the member's path exactly as before. Nothing becomes
statically unreachable and no pyright suppression is added.
TODO.md is a local worklist, swept into the branch's first commit by a
rebase. It was never meant to ship.
@bergundy
bergundy force-pushed the python-dataclasses branch from 3244444 to 52beebb Compare August 20, 2026 20:44
Comment thread src/generator/python.rs
path.push("__init__.py");
let mut contents = String::from("# Generated by nexgen. DO NOT EDIT!\n\n");
let mut wrote_import = false;
let exports_json_runtime = branch.module_path.is_root() && branch_has_json_models(branch);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be leaking json stuff from the backend. I believe the backend is capable of adding additional needed imports/exports in the model file via RenderedModelFragments but this is an extension to be able to add top level init exports. Potentially a separate backend call is also possible but probably less ideal due to shared logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants