Emit Python JSON-Schema models as dataclasses with transfer type converters - #125
Open
bergundy wants to merge 17 commits into
Open
Emit Python JSON-Schema models as dataclasses with transfer type converters#125bergundy wants to merge 17 commits into
bergundy wants to merge 17 commits into
Conversation
bergundy
marked this pull request as draft
August 16, 2026 21:49
bergundy
marked this pull request as ready for review
August 17, 2026 22:07
bergundy
force-pushed
the
python-dataclasses
branch
from
August 19, 2026 21:19
559eb71 to
3244444
Compare
…erters
Generated Python models were the only emitter output requiring a contrib
dependency: every model was a strict `pydantic.BaseModel` carrying its wire
contract as `ConfigDict`, `Field(alias=...)`, `Annotated` validator aliases,
five flavors of `model_validator`, and a `model_serializer` keyed on
`model_fields_set`. That forced `pydantic` on consumers, forced the non-default
`pydantic_data_converter`, and left Python the odd one out on error aggregation.
Models are now plain `@dataclasses.dataclass(slots=True, kw_only=True)` types
whose entire wire contract lives in a generated
`_<Model>TransferTypeConverter`, registered on the class so the *default*
Temporal data converter picks it up with no user wiring. This is structurally
the TypeScript design -- inert data plus an off-model converter owning both
directions -- expressed through the SDK's transfer-type hook. Validation
aggregates into a generated `ValidationError` over `Violation { path, reason }`,
so all four targets now share one error shape.
Two deliberate behavior changes, both keeping Python in line with the other
targets rather than ahead of them:
- Optional+nullable collapses. A dataclass has no presence channel, so absent
and explicit `null` both read as `None` and both re-serialize as omitted.
This extends P1's exception (a) from Go/Java to Python.
- `default` is advisory, as in TypeScript. The field stays `T | None = None`
and the value moves to a module-level `DEFAULT_<FIELD>` constant. Baking the
default into the field would have made Python's wire a superset of the shared
fixtures -- a third exception to round-trip byte-identity, where P1 allows
two. Python trades pydantic's free materialize-on-read for byte-identity.
`additionalProperties` becomes an explicit `additional_properties` member,
matching Go/Java/TS and folding Python into the catch-all name collision that
the loader already rejected for it.
The runtime `_definitions.py` wraps `transfer_type_convertible` in a shim that
erases the converter's value-type parameter: binding it directly on the
decorated class is circular for a static type checker and degrades every model
to `Unknown`. `src/generator/proto/python.rs` has the same latent issue, benign
only because a test imports those models first.
Also corrects pre-existing spec drift found while auditing the Python cells:
the string-length reason was documented as `length must be <= N, got M` for all
four languages where every emitter emits `must have length <= N, got M`;
`required.md` documented a `required property "x" is missing` reason that no
emitter produces; `additionalProperties.md` claimed Python was exempt from the
catch-all collision reject that the loader applies to every language; and
`uniqueItems.md` described a hash-set membership test that raises on the
unhashable dataclass elements Python now generates.
The generated temporal helpers validated with the pinned regex and then handed the string to a `datetime` parser that could still raise, so three classes of input escaped the converter as a bare `ValueError` instead of the aggregated `ValidationError` (P11) -- and the serialize direction had no validation at all, so an unchecked dataclass silently produced wire bytes its own parser rejects (P12). - Year 0000. `datetime.MINYEAR` is 1, so the value has no Python representation; `_valid_temporal_calendar` now rejects it and `_temporal_reason` names the limit. Go/TypeScript/Java all accept year 0, so this is a genuine per-language accept-set divergence Python cannot avoid. - Sub-second precision. Before 3.11 `fromisoformat` parses only the fraction widths `isoformat` writes, so RFC 3339 `.1` and `.1234567` raised on the declared 3.10 floor while every other target accepted them. `_temporal_isoformat` pads or truncates the fraction to 6 digits (and folds in the `Z` rewrite), so the accepted set no longer depends on the interpreter version. Digits past the sixth are dropped at `datetime`'s own resolution -- the loss P1 exception (b) allows, mirroring Go's truncation at nanoseconds. Canonical output is unchanged: `_temporal_frac` re-trims, so `.1` still writes as `.1`. - Duration overflow. CPython refuses `int()` on more than 4300 digits, so the guard now bounds the magnitude by digit count before converting (leading zeros stripped first, matching TypeScript's `Number()`), and a 5000-digit component produces a violation instead of crashing. - Serialize-side representability. `_check_date_time`/`_check_time`/ `_check_duration` hold a materialized value to what the narrowed grammar can spell -- a naive datetime, a sub-minute UTC offset, a negative, sub-second or over-cap duration -- and append a `Violation` under the field's own path rather than emitting bad data. `date` needs no predicate: every `datetime.date` writes a valid wire date. The predicates land in the shared `Validate` layer (P12.2) via `render_py_field_checks`, so the call site stays a single expression and the serialize path picks them up for declared properties, nullable members and typed map members alike, with the correct path and full aggregation.
Two wire-compatibility fixes in the JSON-Schema Python emitter.
`multipleOf` on a `number` crashed instead of validating. Python's
`json.loads` accepts the `Infinity`/`-Infinity`/`NaN` literals its dialect
adds, and `math.fmod(inf, n)` raises `ValueError` (an integer literal past
the binary64 range raises `OverflowError`), so untrusted wire bytes escaped
the aggregated `ValidationError` (P11). A `number` without `multipleOf` was
worse: `inf`/`nan` parsed and re-serialized verbatim, bytes that Go's
`json.Unmarshal`, `JSON.parse` and Jackson all reject (P1). The shared
numeric checks now guard finiteness first and hang the remaining predicates
off its `else`, so the check runs in both directions (P12). This converges on
Go, which rejects every non-finite and out-of-binary64-range `number` on
parse and refuses to marshal one on the way out.
An integer `const`/`enum` kept the wire float: `{"revision": 1.0}` stored
`1.0` and re-serialized as `1.0`, where Go (`parseIntegerField`) and Java
(`SpecNumbers.specLong`) normalize to an integer and emit `1`. A closed
numeric set now routes the wire value through `_parse_spec_integer` before
the membership comparison, which also reinstates the `1.5` reject and the
integer cap those fields bypassed. Float-valued sets have no `Literal` form
(PEP 586) and keep the wire value as it arrived.
The checked-in samples are regenerated separately.
… alias docs
Four fixes to the generated Python JSON-Schema union layer and serialize path.
1. A union's array branch now decodes elementwise (P1). It cast the whole
value to `list[float]` and ran only `minItems`/`uniqueItems`, so any list
was admitted: `{"measurements": ["a", "b"]}` round-tripped verbatim while
Go decodes `[]float64` and Java binds a typed list, both rejecting. The
branch reuses the same element parse a declared array member runs (split
out of `render_array_parser` as `render_py_array_elements`, which the token
guard lets the union call without re-testing `isinstance(value, list)`), so
a bad element reports at its own index (`measurements[0]`). This also
resolves the `True == 1` uniqueness discrepancy at its root: a `bool` in a
`list[float]` is now rejected as a bad element before `_check_unique_items`
sees it.
2. Serialize-side nested violations are re-pathed and aggregated (P11/P12).
`to_transfer_type` wrapped no nested conversion, so a nested
`ValidationError` propagated raw -- discarding the parent's already
collected violations and its own path prefix. Every nested conversion --
`$ref` members, array elements, typed-map members and union dispatchers --
now runs under a `try` that funnels through `_collect`, the analogue of
Go's `mergeNested`: a `Showcase` with a bad `name` and a bad
`location.city` reports both, under those paths, and a bad element reports
at `segments[1]` / `rows[1].cell`.
3. A union's serialize now rejects a value matching no branch (P12). The
dispatch still falls through to its last branch unguarded (that decision
is deliberate and documented), but a value in *no* branch was emitted
verbatim -- bytes every parser rejects. The terminal test widens the value
to `object` first, so the guards are neither provably exhaustive nor
redundant and basedpyright stays at zero errors and zero warnings.
4. A union `TypeAlias`'s docstring follows its assignment, as a module-level
variable docstring must and as dataclass members already do. Emitted
before it, each union's description documented the preceding statement and
the last alias's text was dropped.
An optional+nullable member no longer repeats the caller's `is not None`
guard around its serialize-side checks, which pyright reports as an
unnecessary comparison.
Checked-in samples are intentionally not regenerated here.
A declared property named after one of the converter's own locals silently
disabled validation. `violations: list[Violation] = []` was rebound by the
`violations` property's slot (`violations: str | None = None`), so the collected
violations were thrown away and an invalid payload was returned as a model;
`raw`, `len`, `int`, `str`, `bool`, `dict`, `isinstance`, `typing`, `math` and
`out` crashed every payload instead.
Rather than blocklist names — a property may be named anything — the parse body
now holds each property's value in a `<member>_value` slot local, extending the
discipline the emitter already applied to its temporaries (`{slot}_raw`,
`{slot}_parsed`, …). No fixed local, builtin, imported module, or synthesized
module-level name ends in `_value`, and no derived temporary does either, so the
slot cannot collide with anything: the shadow is structurally impossible instead
of merely unlisted.
The module-level names the emitter synthesizes beyond `DEFAULT_<FIELD>` now
participate in the P15 collision pass, computed through the emitter's own naming
helpers so the check cannot drift from what is emitted:
- `_<MODEL>_DECLARED` — `to_shouty_snake_case` is not injective over verbatim
`x-py-name` overrides, so `ContactPy` and `ContactPY` shared one frozenset and
the loser's declared properties leaked into its catch-all (a P13 break).
- `_<base>_{from,to}_transfer_type` — likewise non-injective, and a named union's
base can coincide with an inline `<model>_<member>` one. An inline union is now
named from the member's *emitted* identifier, so an override moves it (P15's
escape hatch has to reach every synthesized name).
- `_<Model>TransferTypeConverter` and `_PATTERN_<HEX>` — injective in their own
family, but a verbatim override can spell either; the pattern constants are
keyed by pattern text, so identical patterns still share one constant.
- the converter bodies' own locals, for the mirror-image case: a type overridden
to `raw` is shadowed inside every body that parses one.
Also reserves `_definitions` as an input-module name alongside `definitions`:
Python emits its shared runtime as `_definitions.py`, so an input named
`_definitions.yaml` emitted a `_definitions/` package directory at that module's
own import path, shadowing it and breaking every generated
`from .._definitions import ...`.
The checked-in Python JSON samples are left unregenerated; the two snapshot
tests fail until they are rebuilt.
Applies the five review fixes to the checked-in generated output, and records that both `definitions` and `_definitions` are reserved module names now that the loader rejects either spelling for every target. The regenerated diff is dominated by two mechanical changes -- the converter's per-property local is now `<member>_value` rather than the bare member identifier, and the serialize path assigns through statements so a nested conversion can be wrapped -- plus the new finiteness, temporal and closed-value checks.
The Python round-trip suite compared `json.loads(payload.data)` on both sides, so
every assertion ran on a parsed value and nothing verified the wire bytes. That is
the whole justification for one of the design decisions -- a schema `default` stays
off the dataclass field so Python's output is byte-identical to the shared fixtures
-- and it let a real bug through: an integer `const`/`enum` kept its wire `float`,
so `{"revision": 1.0}` re-emitted as `1.0`, and `1 == 1.0` in Python.
`json_converter_helper` now offers `encode_bytes` (the payload's own `data`) against
`canonical_fixture_bytes` (the fixture through `json.dumps(..., sort_keys=True,
separators=(",", ":"))`). Canonicalizing normalizes only insignificant whitespace
and member order -- neither part of the contract -- while preserving the numeric
form, string escaping and the presence of every key. Every per-suite round-trip
assertion moves to that comparison.
The two deviations are one central, documented table rather than a weakened
assertion: `COLLAPSED_NULL_MEMBERS` names the five fixtures carrying an explicit
`null` on an optional+nullable member (P1 exception (a)), and
`NON_CANONICAL_FIXTURES` the one fixture that exists to be normalized. A schema
`default` needs no entry -- an unset defaulted key is omitted, which is *why* the
bytes match. `test_wire_fixtures.py` sweeps all 48 fixtures, fails if one is added
without a model, and asserts each exception entry is both necessary and sufficient.
Per fix, a test that fails without it:
- Integer closed values normalize (`revision`/`tier` from `1.0`/`2.0` re-emit as
`1`/`2`, asserted on bytes -- the parsed comparison passes either way), plus the
`1.5` and integer-cap paths the closed set bypassed.
- Non-finite numbers are rejected in both directions, reached through raw wire text
since `json.loads` accepts `Infinity`/`-Infinity`/`NaN`, plus `1e400` and a
401-digit integer literal.
- Year 0000 is a violation naming `datetime.MINYEAR`; sub-second widths `.1`, `.12`
and `.1234567890` are accepted and re-emit canonically; a 5000-digit duration
component is a violation.
- Serialize-side temporal representability: a naive `datetime`, a negative,
fractional and over-cap `timedelta`, and a sub-minute offset, each under the
field's own path and aggregating.
- Serialize-side nested aggregation: one model with a flat failure plus
`location.city`, `segments[1]`, `rows[1].cell` and `choices.a.kind` reports all
five, fully pathed.
- A union's array branch types every element; a value in no branch is rejected on
serialize rather than emitted.
The identifier-shadowing fix needs a schema no sample declares, so it lands in
`tests/generate_python.rs`: sixteen properties named after the converter's own
locals, builtins, imported modules and method parameters. The rendered output is
asserted for the `_value` slot mechanism, and the generated package is then *run*,
because the failure was silent -- a `violations` property rebound the accumulator
and an invalid payload came back as a model.
Finally the sample suite now also runs on 3.10, the declared floor. The existing
AST check validates syntax only and the project environment is 3.13, so nothing
would have caught the sub-second bug -- it raised on 3.10 alone. `uv run --python
3.10 --locked pytest` reuses the same lockfile in an environment under `target/`,
adding ~3s and no second lockfile to maintain.
…eason Two lint/consistency fixes in the generated Python JSON-Schema layer, plus the gate that let the first one through. 1. A `const`/`enum` check now tests membership in a tuple of the admissible values in both directions. The parse side chained one `!=` per member, so a boolean `const` emitted `enabled_value_raw != True` -- a lint error in the user's repository (ruff E712) and nothing like hand-written Python (P2) -- and a multi-member `enum` emitted one comparison per member. The `!=` chain would likewise emit `!= None` (E711) for any future null-valued set. Parse and serialize now share one shape (`py_value_tuple`), which also drops the magic trailing comma that exploded every multi-member serialize-side test across five lines. A `contains` matcher's `const` takes the same shape, where a boolean matcher would have emitted `element == True`. 2. A mistyped array element now reports the type it failed to be (`tags[0]`, `expected string`), as an element of every other type already did and as Java's element loop reports. A plain `string` element took a special case that reported a bare `expected element`, naming neither the expected type nor anything the element's own indexed path did not already carry; the special case existed only to mirror the TypeScript emitter, whose `expected element` branch already diverges from Python for a *constrained* string element (and drops that element's constraints entirely -- a separate defect, TypeScript being out of scope here). Every element kind now takes the same parse the value in that position takes anywhere else, and `items.md` states the reason convention. `scripts/validate.sh` ran `ruff format --check` but never `ruff check`, so lint defects in generated Python were ungated -- which is how the E712 survived. Both Python tiers now run it. It also surfaced eight pre-existing E402s in the advanced tier's hand-written tests, whose `wit.*` imports sat below two path constants for no reason; they move up with the other imports, matching the sibling test module.
A `string` element schema took a special case in the array parser: a bare
`typeof element !== 'string'` check followed by an assignment, skipping
`render_ts_string_checks` entirely. An
`items: { type: string, minLength: 3, pattern: "^[a-z]+$" }` array therefore
accepted `["a"]` and `["A"]` in TypeScript while Go, Python and Java rejected
both -- an accept-set divergence, which P1 makes part of the wire contract
rather than a per-language detail. The element's compiled `PATTERN_<HEX>`
constant was emitted and never referenced, dead code in the user's repository
and the independent signal that the constraint had been dropped.
Deleting the special case routes a `string` element through
`render_value_parser_at_depth`, the same parse every other element kind (and
every non-element position) already took. That applies the missing
`minLength`/`maxLength`/`pattern`/`format` checks at the element's own index,
and reports a mistyped element as `expected string` rather than the bare
`expected element`, which named neither the expected type nor anything the
indexed path did not already carry. Python removed the identical inherited
special case in e6948ca, so the two languages' element reasons are byte-equal
again -- the standing goal recorded in PRINCIPLES Python §2.
The regenerated samples change only that reason text: no checked-in schema has
a *constrained* string element, which is why the dropped constraints never
surfaced in a sample diff. The showcase suite gains the element-reason
assertion Python's suite already carries.
The rebase put two independent reworks of the same files together, and three seams needed closing: - `DEFAULT_<FIELD>` is now named off the **emitted member identifier** in Python as it already is in TypeScript, so an `x-py-name` override moves the constant with its member (P15) and the loader's shared replication of the rule matches what each generator emits. The loader also derives the identifier per language rather than always through TypeScript's mapping. - The collision test that used two *models* no longer describes a collision: `DEFAULT_<MODEL>_<FIELD>` qualification separates them. It now pins the case qualification cannot resolve — two members of one model — in both languages, plus the override that opens the escape hatch, and keeps the two-model shape as an accepted case. - A TypeScript-only converter-collision test asserted Python accepts the same schema. Python now derives module-level names from the type name too, and `HTTPError`/`HttpError` fold together in `_HTTP_ERROR_DECLARED`, so it rejects for its own reason. Also updates two assertions the dataclass rewrite invalidated (`class Page(` is now `class Page:`) and one showcase test still calling `ShowcaseMapper`.
A union serialize function guards every branch but the last and falls through to it. For a *named* union that was already safe -- the function ran the union's no-branch-matched test first, so the fallthrough was only reached by a value some branch admitted. An inline property-position union had no such test. The declaring converter appended the `expected one of` violation and then handed the same value to the dispatch, so a member holding a value no branch admits reached the last branch's converter and raised whatever its first attribute access raised -- `AttributeError: 'int' object has no attribute 'kind'`, which is not the `ValidationError` the caller catches, so it propagated alone and discarded every violation already collected (P11). Run the checks inside the serialize function for every union, named or inline, ahead of the dispatch: that is the only place that can stop it. The declaring member drops its now-duplicate check and keeps only the `_collect` re-path, which reports the union function's empty-path violations under the member's path exactly as before. Nothing becomes statically unreachable and no pyright suppression is added.
TODO.md is a local worklist, swept into the branch's first commit by a rebase. It was never meant to ship.
bergundy
force-pushed
the
python-dataclasses
branch
from
August 20, 2026 20:44
3244444 to
52beebb
Compare
tconley1428
reviewed
Aug 20, 2026
| path.push("__init__.py"); | ||
| let mut contents = String::from("# Generated by nexgen. DO NOT EDIT!\n\n"); | ||
| let mut wrote_import = false; | ||
| let exports_json_runtime = branch.module_path.is_root() && branch_has_json_models(branch); |
Collaborator
There was a problem hiding this comment.
This seems to be leaking json stuff from the backend. I believe the backend is capable of adding additional needed imports/exports in the model file via RenderedModelFragments but this is an extension to be able to add top level init exports. Potentially a separate backend call is also possible but probably less ideal due to shared logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #124 (
ts-transfer-type-encoding) — the base is that branch, notmain. Merge after #124.Summary
Generated Python JSON Schema models now use slotted, keyword-only dataclasses instead of
pydantic.BaseModel. Their wire contract lives in generated transfer type converters discovered by Temporal's default data converter, so consumers no longer need Pydantic or a custom Pydantic data converter.The generated API now follows these rules:
T | None.del model.fieldrestores the unset state.additional_properties.ValidationError/Violation { path, reason }shape.ValidationErrorandViolationare re-exported from every generated package root, so consumers do not import the private_definitionsmodule.This keeps materialized reads separate from wire presence: an unset default-bearing property reads as its schema default but remains omitted when serialized, while an explicitly assigned value is emitted.
Correctness fixes found during the rewrite
The dataclass conversion exposed several existing cross-language and wire-compatibility gaps. This branch also:
const/enumchecks and keeps generated Python Ruff-clean;ruff checkto the repository validation script.Every generated-output change was made in the generator and regenerated with
cargo build-json-examples --lang python.Review
Best reviewed one commit at a time, starting at
93f55d8. The final generated-default API is concentrated in:4a2ab03— materialize schema defaults through properties;988fc7e— retainT | Noneunion syntax;4cbd8f9— clear explicit values with property deleters;e439348— compact the Unreleased changelog entry; and559eb71— export validation error types from generated package roots.Verification
./scripts/validate.shpasses under the nvm-managed Node 26 runtime. This includes:cargo test --features advancedsuite;Known follow-up
src/generator/proto/python.rsstill emits the baretransfer_type_convertibledecorator for WIT-generated Python. Its circular static typing issue is separate from this JSON Schema rewrite and remains follow-up work.