Boxed any with shape descriptors, structural and tuple patterns

[MelbourneDeveloper]

TLDR

any stops being a bare machine word — it becomes a heap box carrying a shape descriptor, readable only through new structural ({ x, .. }) and tuple ((a, b)) match patterns, with every recovery-by-annotation path now a truthful compile error instead of a printed heap address or a segfault.

Details

New representation for any (crates/osprey-codegen/src/anybox.rs, new, 552 lines)

• An erased value is now a pointer to { i8* desc, i64 payload }. desc points at a per-shape module global @osp.any.desc.<slug> (private constant { i64 kind, i8* render }). Eight descriptor kinds keyed by a DescKey enum (int, bool, float, string, row, union, result, opaque); two erasure sites with the same shape share one global, and that shared pointer identity is what narrowing compares. Slugs are deterministic, so both flavors emit byte-identical IR.
• A record erasure DEEP-boxes through an emitted i8* @osp.any.boxrow.<owner>(i8* src) — one child box per field — so every row with the same field names erases identically regardless of source layout. A union keeps its tagged block (variant decided at run time); a Result erases whole as DescKey::ResultOf(inner); lists, maps, GPU buffers, HttpResponse and any owner with no record/union layout become named opaques (<list>/<map>/<gpu>/<handle>). A record whose row is not a declared record is a hard CodegenError — the backend backstop for the checker's anonymous-record rejection.
• LType::Any is added as a distinct LLVM-i8* type; ltype_of_con maps names::ANY to it instead of LType::I64. New LType::is_managed_ptr() replaces hand-written Str | Ptr matches at eight ownership sites (arc::managed, listlit element kind and to_runtime_list, collections::managed_flag, fiber::gen_spawn result slot, iter::acc_step fold accumulator, result::make_result, lower::gen_cell_store). New ANY_TAG_SPELLING = "any" so container owner tags name an erased element unambiguously.

Ownership fixes (both were live defects, #208/#209)

• Erased boxes are now tracked by the existing ARC/meta machinery: build_box transfers the payload's +1 into the box (or retains a borrowed one) and masks the payload slot PtrManaged exactly when it holds a pointer, so each backend's drop walk releases through the box. This is the ARC: returning a heap value as any releases it in the producing frame — caller reads a dangling pointer, two calls underflow the live-object counter #208 root cause — an erased return could previously neither move its owner out nor be retained.
• List literals keep erased elements boxed: gen_list no longer folds LType::Any into the uniform I64 element word (which turned the box pointer into an integer and marked the slots unmanaged); lit_owner tags an erased-element literal []any and lit_elem decodes any before i8*.
• The effect mailbox marks Any operand slots managed on both the declared and the resolved/erased parameter paths; a captured mut any rebind now releases the box it displaces.

Erasure is one-way (new rejections, user-visible)

• unify::unify_assignable becomes directional: an any actual against a non-variable, non-any expected is cannot recover `T` from an erased `any`: match its structure instead. Anonymous records get cannot erase an anonymous record into `any`: declare its row as a named type first (narrowing selects only among declared rows).
• Checker::reject_erased_operand rejects binary operators, unary operators, field access and indexing on an any, substituting a plausible result type so errors don't cascade.
• reject_reading_erased rejects every pattern form that carries no runtime row test over an any scrutinee — literal, list, type-annotated (s: string =>), constructor (except Success/Error, whose auto-wrap binds the value whole) and nullary-variant-as-binding arms — with <form> cannot read an erased `any`: match its structure instead. Rejected arms still bind their names as any so one mistake doesn't cascade into unknown-identifier errors.
• Previously-accepted programs that are now errors: let s: string = erased(), fn f() -> string = erased(), erased() == "ab", erased().name, and an any-typed parameter used with +.

Structural and tuple patterns

• Pattern::Structural changes from fields: Vec<String> to fields: Vec<(field, binder)> plus open: bool; new osprey_ast::tuple_pattern(binders) builds the positional row (slot i → decimal field name "0", "1", …), so ML (a, b) and Default (a, b) produce identical AST. Consumers updated: freevars, osprey-project rewrite.rs/state_support.rs, effect_rows::bind_pattern (projects by field name, records under the binder). Internal API break for crate consumers.
• Default grammar: the inline '{' field_pattern '}' alternative becomes a named structural_pattern rule with an optional trailing , ..; a new tuple_pattern requires two or more slots, each an identifier or _, so (x) never reaches pattern position. The .. marker lives on structural_pattern, not field_pattern, so the structural ternary and Ctor { … } / p: { x, y } cannot spell an open row. The structural ternary loses its static PREC.ternary in favour of nine newly declared GLR conflicts; the token after } (? vs =>) decides.
• ML frontend: {/} become real lexemes (TokKind::LBrace/RBrace) and count toward layout bracket depth; a new structural_pattern parser reads { a, b } / { a, .. }, and group_pattern no longer rejects commas — a single element still erases to grouping, a comma list becomes MlPattern::Tuple with each slot required to be a binder or _. Consequence: a brace in ML expression position now reports unexpected token LBrace in expression (a parser error) instead of unexpected character '{' (a lexer error); braces still have no ML expression form.
• Checker: bind_structural + structural_row + check_row_selects do real row selection. Over a Type::Record or a Con naming a record constructor, binders take their declared field types; over any every binder is itself any (narrowing is not recovery). New errors: structural pattern names `z`, but T has no such field, a closed structural pattern must name the whole row of T; missing y: add `..` to open it, and a structural pattern needs a record or `any` scrutinee (previously such a scrutinee silently bound fresh variables).
• Exhaustiveness and reachability: a match over `any` is never exhaustive: add a catch-all arm; check_redundant_arms now takes the discriminant and reports an earlier `{ x, .. }` arm already covers this row plus identical closed rows compared as sets (re-spelling field order does not evade it); over a concrete record a structural arm ends the reachable arms.
• Codegen: gen_match routes any match containing a structural arm to a new gen_structural_match, ahead of the union and literal backends (which previously rejected structural arms). An LType::Any scrutinee opens the box once and tests each arm's compile-time candidate rows with icmp eq i8* — no run-time name comparison — binding each field to the row block's child box as Any; a concrete record binds statically off the block; any other representation skips field-naming arms so the catch-all runs. The candidate table is seeded from declared record constructors in compile_program_with_options. finish_phi boxes non-Any arm results when any arm yields a box, so a catch-all's concrete value can't be read as a descriptor.

Rendering

• runtime::to_string_value dispatches LType::Any to a once-emitted @osp.any.to_string(i8* box) that loads the descriptor and calls its render slot. Rows render { x: 1, y: 2 }, unions render Leaf / Node(1, 2) / Circle { radius: 1.0 }, Result renders through its block, opaques render <list>/<map>/<gpu>/<handle>, and a null box renders null instead of faulting. print/toString/interpolation of an any now print the value, not a pointer-sized integer.

New external scanner (tree-sitter-osprey/src/scanner.c, new, 74 lines)

• A tuple arm and a postfix argument list both open with (. The scanner emits one zero-width _call_open_gap token (it calls mark_end before consuming anything) that skips only ' ' and '\t' and succeeds only when the next ( is on the callee's line. The call suffix becomes seq($._call_open_gap, '(', …), keeping calls at PREC.member so 1 + id (2) is still 1 + id(2), while a ( opening the next line belongs to that arm's tuple pattern. Stateless (create returns NULL, serialize returns 0). build.rs compiles it via the existing cc invocation — no new dependency.
• The one spelling given up: a callee whose ( opens the next line is no longer a call. The index [ rule is untouched and still requires strict adjacency.

Docs

• Plan docs/plans/0027-any-erasure-and-recovery.md deleted as done (229 lines) and its README row removed. Spec 0004 [TYPE-ANY] rewritten from "carries no runtime type tag / renders its raw pointer-sized representation" to the boxed, descriptor-rendered, structurally-narrowed model with a status block; [TYPE-ROW] generalised to open/closed rows with a revised unification rule; new [TYPE-RECORD-ANON] and [TYPE-TUPLE]. Spec 0007 gains [PATTERN-STRUCTURAL] and [PATTERN-TUPLE] and drops "runtime narrowing of an any by type is not implemented" / "standalone structural record patterns are not implemented"; spec 0003 rewrites the pattern inventory; spec 0024 replaces "(a, b) is not a tuple pattern and is rejected" with [FLAVOR-ML-TUPLE] and adds [FLAVOR-ML-RECORD-ANON] (documented not implemented); spec 0017 reworks one rationale paragraph only.

No new dependencies. No Cargo.toml or package.json change anywhere in the diff. tree-sitter-osprey/src/parser.c and src/grammar.json are regenerated tree-sitter output and were excluded from review for size — nothing more; src/node-types.json is the same kind of generated output and is included.

Follow-up fixes in this PR (found by auditing the diff against the compiler)

• A structural pattern naming the same field twice is now rejected: the closed-row check compared vector length while the runtime test compares name sets, so { x, x } selected a concrete Pair { x, y } and then MISSED that same value once erased to any. Erasure changing whether an arm matches is the one thing [TYPE-ANY] must never do. bind_structural rejects duplicates before any scrutinee-shaped reasoning, so concrete and erased behave identically, and check_row_selects counts distinct names.
• docs/specs/0003-Syntax.md now states the rule the parser actually enforces: same-line for a call's (, strict adjacency for an index's [. Extending the same-line rule to [ was tried and reverted — list-arm matches are written on one line (match xs { [] => 0 [head, ...tail] => … }), so a same-line [ reads 0 [head as an index and breaks benchmarks/cases/listops/listops.osp. Tuple arms have no single-line usage, which is why ( can afford the looser rule.
• Three spec examples showed syntax that does not parse, each sitting above a status block that correctly said so. They now show spellings that compile: 0007's nested-row/rename example, 0004's let pair: (int, string) = (1, "a"), and 0024's pair : int * string -> string. Every code block in those sections was run through the compiler.
• docs/tuples-branch-review.md deleted — a scratch review artifact whose verdict was stale and whose union_owner claim was not backed by the diff.

Repo hygiene

• Root .gitignore regrouped under section headers, two exact duplicates removed (*.wasm, __pycache__/), and test_output.txt / test_stale_reason.txt / osprey_http_state_levels.db anchored to / so they match only the repo root the corpus writes to. All 86,307 repo paths were run through git check-ignore under both rule sets: exactly one decision changes (tests/regressions/basics/files/test_output.txt stops being ignored, which is the point) and nothing becomes newly hidden.
• Four generated files committed against those rules removed: three examples/wasm/build/*.stdout.txt captures (byte-identical to the tracked examples/wasm/*.expectedoutput goldens they duplicate, sole writer scripts/wasm-smoke.mjs, zero readers) and tests/regressions/basics/files/test_output.txt (written by the test that reads it). Eight scratchpad/ files untracked but left on disk.

How Do The Automated Tests Prove It Works?

Codegen IR assertions (crates/osprey-codegen/src/lib.rs)

• erasure_boxes_with_one_way_ownership (replaces recovering_a_pointer_from_an_erased_word_takes_no_ownership) — an erasing return allocates via @osp_alloc_tagged_noinit with meta 513 (payload word masked managed), stores null into its ARC slot (the referent MOVES in, no second owner), and references @osp.any.desc.string; an any -> any pass-through emits no descriptor reference and retains its borrowed return. Pins ARC: returning a heap value as any releases it in the producing frame — caller reads a dangling pointer, two calls underflow the live-object counter #208 shut from both directions.
• erased_values_render_through_their_descriptor — print("${dynamic()}") emits call i8* @osp.any.to_string(i8* …) and that entry dispatches through a register loaded from the descriptor's render slot, rather than formatting the raw word.
• structural_narrowing_compares_descriptors — a { x, y } arm over an erased Point lowers to icmp eq i8* against @osp.any.desc.row.x.y with no strcmp/osp_string_equals anywhere, and the erasure deep-boxes through @osp.any.boxrow.Point.
• structural_match_on_a_concrete_record_is_static — a (n, _) arm over type Pair = Pair(int, string) compiles to plain load i64, i64* field loads with no @osp.any.desc reference at all: typed code does not pay for the erasure machinery.
• effect_result_parameters_preserve_shape_in_direct_and_resuming_abis (amended) — handler arm registers are now positional %__arm0/%__arm1 instead of source binder names, because a binder spelled entry collided with the entry: block label and clang rejected the module; the Result parameter still travels as its block pointer.

Type-checker unit tests (crates/osprey-types/src/pattern.rs, expr.rs)

• shape_reading_patterns_are_rejected_over_an_erased_scrutinee — literal (0), type-annotated (s: string), list ([a, b]) and variant (Red) arms over an any each produce their own named cannot read an erased \any`` diagnostic.
• structural_fields_of_an_erased_scrutinee_are_erased_themselves — a field bound from { n } over any is itself any, so n + 1 is rejected; narrowing cannot be used as recovery.
• a_match_over_any_requires_a_catch_all — a structural-only match over any reports the never-exhaustive error.
• an_open_row_arm_shadows_every_extension_after_it — { x, .. } before { x, y } is rejected, and { x, y } before { y, x } is rejected as the same row (field order does not make a new row).
• a_closed_structural_pattern_must_name_the_whole_row — over Point, { x } names the missing y, { x, .. } is accepted, { z, .. } reports "has no such field".
• tuple_patterns_bind_positional_record_slots — (n, s) over Pair(int, string) gives each binder its declared slot type, so n + length(s) ?: 0 type-checks.
• structural_pattern_binds_record_fields (rewritten) — an unresolved scrutinee now errors with needs a record or `any` scrutinee where it previously bound fresh variables; the concrete-record case still binds field types.
• an_abandoning_arm_may_not_answer_an_erased_word — converts a previously-passing ok(...) case into a rejection: an any-valued abandoning handler arm in a string region now errors instead of silently handing the caller an erased word.

Parser tests

• crates/osprey-syntax/src/default/expr.rs::a_call_accepts_horizontal_space_before_its_argument_list — id(1), id (1), id (1) all lower to the same Expr::Call; this is the source compatibility token.immediate('(') would have broken.
• …::a_spaced_call_binds_tighter_than_an_operator — 1 + id (2) is Binary(Integer(1), Call), never (1 + id)(2); a GLR fork could not guarantee this because both readings tie on dynamic precedence.
• …::a_newline_paren_opens_a_tuple_arm_not_a_call — in { held, .. } => render (held) followed by (n, s) => n, both arms survive: arm 0 keeps its spaced call, arm 1 is a closed Pattern::Structural with fields ("0","n"), ("1","s").
• …::an_index_bracket_must_touch_its_target_unlike_a_call — pins the asymmetry AND the single-line list-arm match it protects, so a future attempt to unify the two rules fails here rather than in the benchmark corpus.
• crates/osprey-syntax/src/default/lower.rs::lowers_assignment_effects_structural_and_list_patterns (assertion updated) — { name, age } lowers to the pair-shaped closed structural pattern.
• crates/osprey-syntax/tests/ml_elegance.rs::a_tuple_pattern_lowers_to_the_positional_row / structural_patterns_parse_closed_and_open / a_grouped_pattern_holds_one_pattern (rewritten) — ML (a, _) produces the same canonical row [("0","a"),("1","")] the Default spelling does; { heading, .. } is open and { name } closed; (Both a, b) now fails with a tuple pattern slot binds a name or `_` instead of the removed "Osprey has no tuple patterns".

Must-reject fixtures (examples/failscompilation/, byte-exact goldens)

• any_recovery_by_annotation.ospo — 4 line:col-anchored errors at 11:3, 12:3, 14:0, 15:0, all cannot recover `T` from an erased `any`, covering a return annotation over a heap payload, a return annotation over a scalar, let viaLet: string and let viaInt: int. These are exactly the forms that previously printed a heap address as a decimal integer (let x: string = <any-typed expr> drops the annotation and prints the pointer as a decimal integer — the same recovery through a function return works #209) or segfaulted with no message.
• any_narrowing_rules.ospo — 10 exact diagnostics in one file: literal arm, s: string arm, variant arm, missing catch-all, unreachable match arm: an earlier `{ x, .. }` arm already covers this row, duplicate structural field in BOTH erased and direct position, cannot apply `==` to an erased `any`, field access on an erased value, and 61:0: let `viaAnon`: cannot erase an anonymous record into `any`.
• ml_any_recovery_by_annotation.ospo — 3 errors at 12:0, 15:0, 18:0: an ML signature line is an annotation and cannot recover a value out of any either.
• ml_any_narrowing_rules.ospo — 7 errors (the duplicate-field pair included); the ML twin reports the variant arm as a constructor pattern (same arm, different frontend lowering), and the annotation-pattern, field-access and anonymous-record cases have no ML surface.
• ml_brace_record_and_question_sigil.ospo re-pinned — the golden's lexer errors 12:8: unexpected character '{' / 12:23: unexpected character '}' become parser errors 13:8: unexpected token LBrace in expression / 13:23: unexpected token RBrace in expression; crates/osprey-cli/tests/examples_compile.rs ML_NEGATIVES is updated to the new substring.

End-to-end corpus (tests/regressions/basics/types/any_type_comprehensive.test.osp / .ospml, one shared golden, run under every memory backend and again on wasm32)

• describeAny selects { held, .. } / { value } / (n, s) / catch-all correctly for a named record, a positional record, a scalar and an erased string (held=dynamic, value=via row, pair=7:seven, other=42, other=stillerased), and reads a runtime-built string back out of an erased row — the recovery annotations lost.
• Ownership at every representation boundary: a flat list literal (other=boxed|other=7), a runtime List append/get (Success(appended)), a fiber result slot (other=fiber), a fold accumulator (other=seed), an effect mailbox operand (auditTally() == 18), and a captured mut any rebind inside a handler (swapped() == "other=11" — the leak the branch review filed as P1). A built-but-never-read erasure is discarded as a statement so the ARC live-object oracle must return to zero.
• describeTwice proves narrowing is repeatable on the same box (held=twin/held=twin); toString(erasedPoint()) == "{ x: 1, y: 2 }" proves descriptor-driven rendering. 13 new assertions in the existing checkAll case; the shared golden gains exactly two stdout lines, narrowed: held=printed and rendered: { x: 1, y: 2 }.
• The change forced two source edits that are themselves evidence: forwarded() -> string became forwardedHeap() forwarding into another any sink, and the ML twin's processAnyValue : any -> any became int -> any, because an any parameter can no longer be an operand of +.

Verification run on this branch

• cargo fmt --all --check: clean.
• cargo clippy --workspace --all-targets -- -D warnings: clean.
• cargo test --workspace: passes, excluding wasm::tests::build_and_run_end_to_end_when_toolchain_present, which fails only because the installed Node is 22.22.2 and it requires Node 24+.
• crates/run_test_corpus.sh under default, gc and arc: 179/179 suites, 179/179 byte-exact goldens, 18/18 alt GPU lowering, and TEST_CORPUS_ARC_LEAKY=0 under arc.
• npx tree-sitter test: passes.

Closes #175. Closes #208. Closes #209.