Surface checks for generated SDKs (design + runnable demo)

.chronus/changes/fix-typeddict-single-overload-2026-6-26-12-56-0.md

@@ -0,0 +1,7 @@
+---
+changeKind: fix
+packages:
+  - "@typespec/http-client-python"
+---
+
+Fix invalid lone `@overload` generated for body parameters in `models-mode: typeddict`. When the binary and JSON overloads are omitted, the single remaining body variant is now emitted as a plain parameter instead of a single `@overload`, which mypy rejects with "Single overload definition, multiple required".

.chronus/changes/python-addTypedDict-2026-3-21-17-47-3.md

@@ -0,0 +1,8 @@
+---
+# Change versionKind to one of: internal, fix, dependencies, feature, deprecation, breaking
+changeKind: feature
+packages:
+  - "@typespec/http-client-python"
+---
+
+[python] Always generate `TypedDict` typing hints for input models in the `types.py` file, and named union aliases in the `_unions.py` file

.chronus/changes/spector-surface-doc-2026-7-7-14-53-0.md

@@ -0,0 +1,7 @@
+---
+changeKind: feature
+packages:
+  - "@typespec/spector"
+---
+
+Add a `@surfaceDoc` decorator and `listSurfaceDocs` for describing, in plain natural language, expected properties of the generated SDK surface (naming, access, inheritance reshaping, operation location, paging, ...) that wire tests can't observe. Authors write the sentence and apply the normal decorator; `listSurfaceDocs` derives the machine-checkable, routable fields from the element's own decorators (`@list`, `@clientName`, `@access`, `@clientLocation`, `@hierarchyBuilding`), falling back to AI verification of the prose when none is recognized. An optional explicit `check` argument lets authors supply a category or routing detail the derivation can't infer. The new `tsp-spector generate-surface-checks` command precomputes a language-neutral `surface-checks.json` manifest from these annotations, analogous to how `@scenarioDoc` feeds the scenario manifest.

.chronus/changes/spector-surface-doc-http-client-python-2026-7-7-14-53-0.md

@@ -0,0 +1,7 @@
+---
+changeKind: internal
+packages:
+  - "@typespec/http-client-python"
+---
+
+Add the emitter-side client-surface verification tooling (`verify.py` and the surface-checks context/prompt assets) that consumes the spector-generated `surface-checks.json` manifest to validate the generated Python SDK surface.

packages/http-client-python/.github/prompts/check-surface.prompt.md

@@ -0,0 +1,66 @@
+---
+description: "Orchestrate surface checks: one parallel deterministic batch, then AI for the rest"
+mode: "agent"
+---
+
+You are the **orchestrator** for surface checks on an emitter's generated
+SDK — properties (naming, visibility/access, shape) the wire tests can't see.
+You verify every check in **two parallel phases**, not one-by-one:
+
+1. **Deterministic phase (parallel, no AI).** One batch call runs every check
+   whose category has a routine — concurrently.
+2. **AI phase (only the leftovers).** You verify the few checks with no
+   deterministic routine; dispatch them in parallel if there are several.
+
+Read:
+
+- `packages/${input:emitter}/eng/scripts/client-criteria/context.md` — Emitter
+  facts (language, generated-root, flavors, checks-doc path) and concept prose.
+- The **checks doc** — the `checks-doc` Markdown table named in `context.md`
+  (default `surface-checks.md`): the batch runner reads it for deterministic
+  checks and you read its rows' `doc` prose for the AI ones. Use this path unless the caller
+  explicitly overrides it.
+- `packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json` — the
+  routing table: a category listed here is deterministic; one not listed is AI.
+
+Inputs:
+
+- Emitter: ${input:emitter:package folder, e.g. http-client-python}
+- Flavor: ${input:flavor:one of the flavors in the emitter's context.md}
+- Scenario: ${input:scenario:optional substring filter on the check id}
+
+## Phase 1 — deterministic batch (parallel)
+
+Run the batch tool **once**. It runs all deterministic checks in parallel and
+prints `{"results": [...], "needs_ai": [...]}`:
+
+```
+python packages/${input:emitter}/eng/scripts/client-criteria/verify.py --batch \
+  --checks packages/${input:emitter}/eng/scripts/client-criteria/<checks-doc from context.md> \
+  --verifiers packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json \
+  --generated-root <generated-root from context.md> \
+  --flavor ${input:flavor} --language <language from context.md> --workers 16
+```
+
+Take `results` as-is — each is a deterministic verdict (`how: deterministic`).
+Do not re-check or second-guess them.
+
+## Phase 2 — AI verification (the residual, in parallel)
+
+For each item in `needs_ai`, verify it yourself: find the target in the generated
+package under `<generated-root>/<flavor>/` (folder matching the scenario), judge
+against its `verify` text and context.md, and cite the exact file + line. Mark
+`how: ai`. **If there are several, dispatch them as parallel subagents** (one per
+item) rather than serially, then collect the verdicts.
+
+## Phase 3 — merge
+
+Combine `results` + your AI verdicts into one table, filtered by the scenario
+input if given:
+
+`id | category | result (PASS/FAIL/N/A) | how (deterministic/ai) | evidence`
+
+End with a summary: pass/fail counts, and how many were code vs AI.
+
+The point: the deterministic bulk is one parallel call decided by code; AI is
+spent only on the checks that actually need judgment.

packages/http-client-python/eng/scripts/client-criteria/README.md

@@ -0,0 +1,133 @@
+# Client surface checks for generated SDKs
+
+_Verify properties of generated SDKs that the wire tests can't observe —
+described once in the spec, precomputed centrally, checked per emitter. Mostly
+deterministic; AI only for the fuzzy residual._
+
+Full design write-up: https://gist.github.com/iscai-msft/8be1b3ecf016bfd3bfbf3a7f6857d78c
+
+## What this is
+
+Spector verifies emitter parity on the **wire**. A large class of behavior never
+reaches the wire — whether an internal type is actually hidden, a property was
+renamed, a method relocated, a model flattened. Today those are checked by hand,
+per language.
+
+**Client surface checks** close that gap. You annotate a spec element with
+`@surfaceDoc("…", "<category>")` — a language-agnostic description of what
+should be true of the generated surface. A precompute step collects every
+`@surfaceDoc` into a shared **checks doc**. Each emitter supplies a
+**context.md** (what those concepts look like in its output) and an optional
+**verifiers.json** (mechanical routing table). A deterministic runner
+(`verify.py`) checks everything with a verifier; only fuzzy items go to the
+`/check-surface` orchestrator prompt.
+
+Whether a check is deterministic or AI is decided by whether the language
+supplied a verifier for its category — not by the scenario. Add a verifier and
+the same check is promoted from AI to deterministic; the spec assertion never
+changes.
+
+## The pieces
+
+| Piece                                                  | Lives in                               | AI?                      |
+| ------------------------------------------------------ | -------------------------------------- | ------------------------ |
+| `@surfaceDoc` decorator                                | spector (with `@scenario`)             | no                       |
+| checks doc (precomputed manifest)                      | generated by spector, ships with specs | no                       |
+| `context.md` + `verifiers.json` (concept descriptions) | each emitter repo                      | no (authored)            |
+| `verify.py` (deterministic runner)                     | shared                                 | no                       |
+| `/check-surface` orchestrator                          | monorepo `.github/prompts/`            | AI only for the residual |
+
+Work is pushed left so AI is the exception: (1) the `@surfaceDoc` manifest
+walk is deterministic; (2) concept descriptions are authored once per language,
+~a dozen entries reused across all scenarios; (3) `verify.py` decides every item
+with a verifier; (4) AI handles only the residual, cached on
+`(item id + generated-file hash + model version)` and scoped to changed packages.
+
+## Running it
+
+One call (Copilot Chat agent mode or CLI):
+
+```
+/check-surface   →   emitter: http-client-python   flavor: azure   scenario: (blank = all)
+```
+
+The prompt is an **orchestrator**: in one agent session it walks the checks doc
+and, per check, consults `verifiers.json` — if the category is listed it runs
+`verify.py` and uses that verdict, otherwise it verifies the check itself (AI).
+It prints one merged table marking each row `deterministic` or `ai`.
+
+### Runnable demo (two examples)
+
+`demo-checks.json` is a self-contained manifest with the two gist examples,
+checked against the real generated Python SDKs under `tests/generated/azure`:
+
+| id                                 | category    | how                                             |
+| ---------------------------------- | ----------- | ----------------------------------------------- |
+| `Client_Naming_UnionEnum_name`     | `naming`    | deterministic (`identifier_idiomatic_casing`)   |
+| `Client_Hierarchy_Dog_extends_Pet` | `hierarchy` | **AI by default** → deterministic once promoted |
+
+Run the deterministic batch (naming passes; hierarchy lands in
+`needs_ai`):
+
+```bash
+python eng/scripts/client-criteria/verify.py --batch \
+  --checks eng/scripts/client-criteria/demo-checks.json \
+  --verifiers eng/scripts/client-criteria/verifiers.json \
+  --generated-root tests/generated --flavor azure --language python
+```
+
+Then rerun with `verifiers.promoted.json`, which adds the one `hierarchy`
+entry — the same hierarchy check flips from AI to a deterministic PASS with no
+change to the spec assertion:
+
+```bash
+python eng/scripts/client-criteria/verify.py --batch \
+  --checks eng/scripts/client-criteria/demo-checks.json \
+  --verifiers eng/scripts/client-criteria/verifiers.promoted.json \
+  --generated-root tests/generated --flavor azure --language python
+```
+
+### Negative fixture (seeing red)
+
+`demo-checks.negative.json` mirrors each example with **one wrong field** (wrong
+casing kind, wrong name, flipped client, wrong base) so the runner must report
+`fail`. Run it with the promoted verifiers so all four resolve deterministically:
+
+```bash
+python eng/scripts/client-criteria/verify.py --batch \
+  --checks eng/scripts/client-criteria/demo-checks.negative.json \
+  --verifiers eng/scripts/client-criteria/verifiers.promoted.json \
+  --generated-root tests/generated --flavor azure --language python
+```
+
+Every item comes back `status: fail` with evidence explaining why, e.g.:
+
+| id                      | why it fails                                                                                                       |
+| ----------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `NAMING_wrong_casing`   | expects the enum as a `property` → snake_case `client_extensible_enum`, which doesn't exist (only PascalCase does) |
+| `NAMING_wrong_name`     | right casing, but `TotallyMadeUpEnum` is not generated                                                             |
+| `LOCATION_wrong_client` | `deleteUser` is missing from the claimed client and still on the one it supposedly left                            |
+| `HIERARCHY_wrong_base`  | generated base is `Dog(Pet…)`, not a subtype of `Animal`                                                           |
+
+PASS and FAIL come from the same routine reading the same files — a wrong
+expectation flips the verdict.
+
+## Onboarding a language
+
+One required file: **context.md** under
+`packages/<emitter>/eng/scripts/client-criteria/` — Emitter facts (language,
+generated-root, flavors, checks-doc path) plus a **concept description** per
+category, written concretely enough to execute as a file check. Optionally add
+**verifiers.json** (routing table mapping categories to `verify.py` routines)
+for a zero-AI path in CI. A new emitter writes no assertions and no per-scenario
+criteria — the decorator, checks doc, and orchestrator are all shared.
+
+## Open questions
+
+- Where does the precomputed checks doc ship — a consumable asset in spector, or
+  alongside the specs in http-specs?
+- Should the manifest record the **resolved per-language symbol name** (so
+  renamed-and-access-controlled types stay mechanically checkable)?
+- Should `verifiers.json` carry executable patterns (glob/regex) so a fully
+  non-AI runner can execute them, with the prompt purely as the fuzzy fallback?
+- How do items mark **N/A** for a language whose surface has no such concept?

packages/http-client-python/eng/scripts/client-criteria/context.md

@@ -0,0 +1,73 @@
+# Client surface checks — context: http-client-python
+
+## Emitter facts
+
+_The shared `/check-surface` prompt and `verify.py` read these._
+
+- **language:** python _(selects which per-language client name applies when a check is language-scoped)_
+- **generated-root:** `packages/http-client-python/tests/generated/<flavor>`
+- **flavors:** `azure`, `unbranded`
+- **checks-doc:** `packages/http-client-python/eng/scripts/client-criteria/surface-checks.md`
+<!-- Markdown table generated from `@surfaceDoc`; both human- and machine-readable (parsed by verify.py). -->
+
+### What differs by flavor (branded = azure, unbranded)
+
+Only the **generated-root** subfolder and the **import namespace** (`azure.core`
+vs `corehttp`). Symbol locations and every concept description below are
+flavor-invariant.
+
+## What each check looks like
+
+What each verifiable concept looks like in generated Python. Authored once,
+reused across every scenario that uses the concept. Prefer concrete, checkable
+verifiers so the deterministic runner can decide **without calling AI** — see
+`verifiers.json` for the machine-readable form.
+
+### access: internal
+
+- **enum** → the class is NOT re-exported from `models/__init__.py`; it may also
+  be defined with a leading `_`.
+- **model** → absent from the package's public `__init__.py` exports.
+- **operation / method** → prefixed `_`, or not present on the public client.
+
+### access: public
+
+- present in the public `__init__.py` exports, no leading underscore.
+
+### naming (`@clientName`)
+
+- the generated identifier equals the client name **recast to Python's idiomatic
+  casing for the symbol kind** (case-sensitive): `enum`/`model`/`type` →
+  `PascalCase`, `property`/`parameter`/`operation` → `snake_case`, enum value →
+  `UPPER_SNAKE`. So an enum renamed to `ClientExtensibleEnum` must appear exactly
+  as `ClientExtensibleEnum` (not `client_extensible_enum`). The per-kind casing
+  map lives in `verifiers.json`; the manifest item carries the symbol `kind`
+  under `details`.
+- **resolving the expected name:** if the check has a `client_names` map
+  (`{"python": "...", "csharp": "..."}`), use this language's value; otherwise
+  `details.name`. A language with no entry is `N/A`.
+
+### exactName (`@exactName`)
+
+- the identifier equals the spec name **byte-for-byte**, no casing transform.
+
+### flatten (`@flattenProperty`)
+
+- the nested model's fields appear directly as attributes on the parent class.
+
+### client-location (`@clientLocation`)
+
+- an operation group is a class under `operations/_operations.py`; a method is a
+  public `def` on it; the **root client** is the main `*Client` class. An
+  operation is "on" a client/group when its snake_case name is a method of the
+  matching class. Deterministic: assert the method is present on the
+  **expected** client/group and absent from the one it moved from.
+
+### hierarchy (`@hierarchyBuilding`)
+
+- a client subtype is expressed as the base in `class <Sub>(<Base>, …)` in
+  `models/_models.py`; inherited members are attributes present on the subclass.
+
+> When an assertion has no concrete verifier here (and none in verifiers.json),
+> `/check-surface` may fall back to judgment — but add a verifier when
+> you can, to keep AI calls rare.

packages/http-client-python/eng/scripts/client-criteria/demo-checks.json

@@ -0,0 +1,20 @@
+{
+  "_comment": "Demo manifest for the two gist examples, checked against the real generated Python SDKs under tests/generated/azure. Item 1 (naming) has a verifier (deterministic); item 2 (hierarchy) has none by default so it routes to AI -- run with --verifiers verifiers.promoted.json to see it become deterministic.",
+  "items": [
+    {
+      "id": "Client_Naming_UnionEnum_name",
+      "scenario": "client-naming",
+      "category": "naming",
+      "target": "ServerExtensibleEnum",
+      "details": { "name": "ClientExtensibleEnum", "kind": "enum" }
+    },
+    {
+      "id": "Client_Hierarchy_Dog_extends_Pet",
+      "scenario": "hierarchy-building",
+      "category": "hierarchy",
+      "target": "Dog",
+      "doc": "Dog is surfaced as a subtype of Pet (inherits Pet's members), not a direct subtype of Animal",
+      "details": { "base": "Pet" }
+    }
+  ]
+}

packages/http-client-python/eng/scripts/client-criteria/demo-checks.negative.json

@@ -0,0 +1,34 @@
+{
+  "_comment": "Negative fixture: deliberately-wrong expectations that MUST fail, so the demo shows red as well as green. Run with --verifiers verifiers.promoted.json so all four run deterministically. Each item mirrors a demo-checks.json item with one wrong field (casing kind, name, client, or base).",
+  "items": [
+    {
+      "id": "NAMING_wrong_casing",
+      "scenario": "client-naming",
+      "category": "naming",
+      "target": "ServerExtensibleEnum",
+      "details": { "name": "ClientExtensibleEnum", "kind": "property" }
+    },
+    {
+      "id": "NAMING_wrong_name",
+      "scenario": "client-naming",
+      "category": "naming",
+      "target": "ServerExtensibleEnum",
+      "details": { "name": "TotallyMadeUpEnum", "kind": "enum" }
+    },
+    {
+      "id": "LOCATION_wrong_client",
+      "scenario": "client-location-move-to-existing-sub-client",
+      "category": "client-location",
+      "target": "deleteUser",
+      "details": { "client": "UserOperations", "absentFrom": "AdminOperations" }
+    },
+    {
+      "id": "HIERARCHY_wrong_base",
+      "scenario": "hierarchy-building",
+      "category": "hierarchy",
+      "target": "Dog",
+      "doc": "Dog is surfaced as a subtype of Animal directly",
+      "details": { "base": "Animal" }
+    }
+  ]
+}