Skip to content

ML flavor front end, formatter, effects, modules, and docs/website overhaul#165

Merged
MelbourneDeveloper merged 64 commits into
mainfrom
prototyping
Jul 1, 2026
Merged

ML flavor front end, formatter, effects, modules, and docs/website overhaul#165
MelbourneDeveloper merged 64 commits into
mainfrom
prototyping

Conversation

@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

TLDR

Adds a second ML (offside/curry-by-default) surface to Osprey — parser, lexer, lowering, formatter, LSP, and VSCode support — proven byte-for-byte equivalent to the Default flavor, alongside a full spec/docs/website overhaul and a browser WebAssembly SQL demo.

Details

ML flavor frontend (crates/osprey-syntax/src/ml/) — a complete indentation-delimited, curry-by-default front end that lowers to the same AST as the Default flavor:

  • lexer.rs / token.rs — offside-rule layout lexer (INDENT/DEDENT, newline-terminated strings).
  • parser.rs / cst.rs — ML grammar: whitespace application (f x), \x => e lambdas, := mutation, = binding, layout blocks — no fn/let/braces.
  • lower.rs (1730 LOC) — lowers the ML CST to the shared AST.
  • crates/osprey-syntax/src/default/ — the existing brace-flavor parser moved under default/ so the two surfaces sit side by side; strings.rs factored out as shared interpolation lexing.
  • .osp → Default, .ospml → ML, selected by extension; a flavor marker conflict is a hard error.

Formatter (crates/osprey-fmt/ + osprey fmt) — new crate with brace (brace.rs) and layout (layout.rs) formatters plus a shared scanner; wired into the CLI as osprey fmt (stdin/dir-recursive/check/rewrite modes) and into make fmt.

Algebraic effects / first-class handlerscompiler/runtime/effects_runtime.c (+250) and osprey-codegen/src/effects.rs (+515) extend effect resumption and make handlers first-class values.

Modules & namespaces — spec 0025 plus supporting type-checker and lowering changes.

LSP / VSCodeosprey-lsp gains flavor-aware diagnostics/features; new osprey-ml.tmLanguage.json, ML snippets, and language-configuration-ml.json.

Website & docs — every website/src/spec/* chapter revised; new chapters 0023 Language Flavors, 0024 ML Flavor Syntax, 0025 Modules and Namespaces; mirrored under docs/specs/. /docs/ index gains an inline Flavors explainer (one language, two ways to write it). New /wasm/ page: an Osprey program compiled to wasm32-wasip1 seeds a real in-browser SQLite (sql.js) database with a tabbed query/add-data/source UI, and both the Osprey and SQL are Prism-highlighted (js/wasm-studio.js). Playground gains a Default⇄ML flavor toggle.

Examples — most examples/tested/* programs gain a byte-identical .ospml twin; several .osp files were de-annotated to lean on inference.

Cleanup note: this branch also carries scratchpad/ working files and */.claude/settings.local.json that should probably be dropped before merge.

How Do The Automated Tests Prove It Works?

make ci (lint + test + build) passes end-to-end, exit 0. Specific coverage:

  • cross_flavor_ir_equiv.rsevery_ml_example_has_a_default_twin and ml_and_default_twins_emit_identical_ir compile each .ospml/.osp pair and assert the emitted LLVM IR is identical, proving the ML surface lowers to the same program.
  • cross_flavor_equiv.rsml_multiparam_equals_default_explicit_curry / ml_uncurried_tuple_equals_default_multiparam pin ML currying semantics against the Default flavor.
  • corpus.rsdefault_examples_format_idempotently and ml_examples_format_idempotently prove osprey fmt is a fixpoint over the whole example corpus (format twice == format once).
  • cli_e2e.rsfmt_*, flavor_marker_conflict_exits_two, check_*, and llvm_emits_ir_and_rejects_ill_typed exercise the fmt/check/llvm CLI surface end-to-end including error exit codes.
  • no_needless_main.rsno_example_wraps_a_trivial_program_in_main enforces the style rule across examples.
  • ml_coverage.rs (823 LOC) — parser/lowering unit coverage for the ML front end.
  • Differential harness (diff_examples.sh / diff_wasm_examples.sh) — every .osp/.ospml in examples/tested/ runs and must match its .expectedoutput byte-for-byte.
  • Website E2E (website/tests/interactions.spec.js)wasm demo seeds a browser SQLite db and runs queries boots the wasm module, asserts #banner reaches "Database ready", the query tab auto-renders a result table with highlighted SQL tokens, the source tab shows highlighted Osprey, and adding a row then re-querying reflects the new data; playground flavor toggle swaps the sample between .osp and .ospml verifies the Default⇄ML toggle.
  • VSCode extension suite — 78 passing, coverage 90.94% (≥89% threshold).

Baseline commit of the in-flight tracing-GC work that was already present in
the working tree (memory_gc.c, osp_gc_shim.h, --memory=gc flag, GC archive
build). Snapshotted as-is so the WebAssembly feature work can layer cleanly on
top; this commit is not part of the wasm change.
…targets

The emitted IR baked in two host C type widths that differ on wasm32 (ILP32):

- string length called libc strlen, declared i64, but size_t is 32-bit on
  wasm32 -> a wasm-ld signature mismatch that traps at runtime. Route both the
  `length` builtin and string concatenation through a new runtime shim
  osp_strlen(const char*) -> int64_t, so the size_t->int64 cast lives in C
  (correct per target) and the IR stays i64 on every target.
- int-to-string used sprintf("%ld", i64). On wasm32 `long` is 32-bit, so big
  ints printed wrong. Use "%lld" (long long is 64-bit everywhere); identical
  to %ld on LP64 native.

Native output is byte-identical: 48/48 differential golden tests pass.
…ssembly

Add `--target=wasm32` (and `-o <out>`) to the osprey CLI: lower the existing
target-agnostic IR to wasm32-wasip1 and link a command module with wasm-ld.

- crates/osprey-cli/src/wasm.rs: the wasm backend driver. Appends a __main_void
  entry thunk (sidesteps the clang/wasi-libc main-mangling skew), lowers IR to a
  wasm object with `clang -c` (so clang never demands the absent
  libclang_rt.builtins-wasm32.a), then drives wasm-ld directly: crt1-command.o +
  object + the portable runtime archive + libc. Sysroot/tool discovery via env
  overrides + conventional locations. Pure helpers are unit-tested.
- Makefile: `make wasm` builds libosprey_runtime_wasm.a from the portable C
  subset (no pthreads/sockets/OpenSSL/syscalls) and compiles the example;
  `make wasm-test` validates + smoke-runs it.
- examples/wasm/: hello.osp, a self-contained browser loader with an inline WASI
  shim (fd_write -> console), README, expected output.
- scripts/wasm-smoke.mjs: validate + run under Node's built-in WASI, assert stdout.

[WASM-TARGET]
…RGET]

Add deterministic, toolchain-free unit tests for the wasm32 CLI driver
(run_tool/compile_object/link/run_host/build/run, stubbing clang/wasm-ld/
wasmtime via OSPREY_WASM_* + a temp sysroot + a discoverable stub archive)
and the --memory / --debug / --target=wasm32 dispatch paths in main.rs.

Restores osprey-cli to the 95% coverage floor (was 89.9%) without relying
on a wasm toolchain being present in the coverage job.
Add the generated function-reference pages for the CSPRNG builtins and
link them from the function index.
…00% [DEBUGGER]

Add the Osprey debug panel (stack frames, locals, program details) wired into
the activity bar, a comprehensive real-lldb-dap VSIX e2e suite (step
over/in/out, continue, conditional/multi breakpoints, watch/evaluate,
scopes/variables) on a shared DAP test harness, and drive osprey-debug to
100% line coverage. Full VSIX suite 76 passing; extension coverage 90.84%.
- cargo fmt the new osprey-cli wasm unit tests (rust job Format check diff).
- wasm job: the wasi-sdk-24 sysroot tarball unpacks to a versioned top dir
  (wasi-sysroot-24.0/), so 'test -d .../wasi-sysroot' failed silently. Extract
  with --strip-components=1 into a stable path and assert lib/wasm32-wasip1.
- setup-osprey-compiler: install lldb and expose lldb-dap at /usr/bin so the
  VS Code debugger E2E resolve it and run in CI instead of failing 'not found'.
…UGGER-LLDB-DAP-VERSION]

The VS Code debugger E2E suite failed on ubuntu-latest: its distro lldb-dap
(LLVM 18) DAP server mis-resolves a stopped frame's source path to "." and
ignores breakpoint conditions on x86_64. liblldb 18 reads the osprey-emitted
DWARF correctly (confirmed via lldb CLI and a direct DAP client), and lldb-dap
22 on macOS passes the whole suite — the defect is the old DAP server, fixed
from lldb-dap 19. Install lldb-dap 20 from apt.llvm.org in the compiler setup
action and pin /usr/bin/lldb-dap to it; clang/llvm 18 still builds the C runtime.

Expand the VSIX breakpoint E2E so the same test asserts F10 / Step Over executes
a call (bump) without descending into it — a regression guard for "step over
behaved like step in". [DEBUGGER-STEP-OVER]

Makefile: collapse vsix-rebuild-reinstall to a single `code --install-extension
--force` (drop the separate uninstall) so a running VS Code reconciles its
extension host once, and id-scope the vsix glob to osprey-*.vsix.
…an [DEBUGGER-DBG-DECLARE]

An inline `@llvm.dbg.value` for a `let` binding lowers to a DBG_VALUE whose
line-table row is line 0; between two statements that leaves a stray line-0
entry. lldb-dap reads the osprey DWARF fine on most toolchains, but on x86_64
that stray row derails breakpoint line resolution — a frame at the print line
or a conditional hit reports `line 0`, so `assertCurrentLine` sees "stop on
line 0". (Verified against lldb-dap 20 on x86_64/CI and arm64, and lldb-dap 22
on macOS.)

Emit locals with `@llvm.dbg.declare` over a dedicated, debug-only stack slot
instead: the variable lives at a fixed frame offset (readable at every PC in
scope) and no inline DBG_VALUE means no stray line-0 row. Parameters keep
`@llvm.dbg.value` — they are SSA arguments live from entry, so they read
correctly even at a conditional breakpoint on the function's first line (a slot
store would not have run yet there), and at function entry they never introduce
an inter-statement line-0 row.

Verified end to end: all debugger E2E tests pass on lldb-dap 22 (macOS) and the
previously-failing conditional/scopes/line-6/step scenarios resolve to the
correct lines with readable locals on lldb-dap 20.
…DEBUGGER-STEP-OVER]

lldb-dap can briefly report a freshly-stopped frame's source path as "." (and
its line as 0) on the first `stackTrace` of a session, before the module's line
table finishes resolving, then correct both. The osprey DWARF is clean — the
frame resolves to the real file on a subsequent request (verified: arm64
lldb-dap 20 and macOS lldb-dap 22 always resolve; only x86_64 cold starts race)
— so this is a poll race, not a debug-info defect.

`waitForStop` now keeps polling for a top frame whose source path is resolved
(not ".") on a real 1-based line, holding the first unresolved stop only as a
fallback returned once a short grace elapses. A frame that genuinely has no
source (the C entry stub a stopOnEntry launch pauses in) still returns via that
fallback, so nothing hangs. Assertions are unchanged — this only stops sampling
the cold-start transient.
…LDB-DAP-VERSION]

lldb-dap 18 (ubuntu-latest distro) mis-resolves source paths and ignores
breakpoint conditions on x86_64; lldb-dap 20 fixed those but flakes on x86_64
step/line resolution (a different debugger E2E test failed each CI run, with
"custom request failed" timeouts). lldb-dap 22 (LLVM 22.1.8) is the exact
toolchain developers run on macOS, where the suite passes deterministically;
verified stable on Linux too (all scenarios + step-over correct, repeated).
apt.llvm.org ships lldb-22 for noble, so install that and pin /usr/bin/lldb-dap
to it. The dbg.declare line-table fix and the harness's source-resolution wait
remain as version-independent robustness.
- vsix-rebuild-reinstall: drop the redundant `code --uninstall-extension`
  step and rely on `--install-extension --force` (idempotent in-place
  upgrade) so the running VSCode reconciles its extension host once, not
  twice; remove the now-dead _vsix_uninstall target.
- Pin both .vsix globs to osprey-*.vsix (clean + install) so no non-osprey
  extension is ever referenced.
- Add `make wasm-serve`: builds via `make wasm`, static-hosts examples/wasm
  at localhost:8080 and opens the browser. `make wasm` stays headless for CI.
- CLAUDE.md: add act-autonomously directive (user edit).
Replace the 3-line width-fix snippet with a one-screen tour of the wasm-safe
heart of the language, every value really computed:
- algebraic effects: one pure `account()` run in two worlds (real stateful
  handler vs frozen mock ledger) — meaning chosen by the handler
- union type + exhaustive pattern matching (Tier/badge)
- Hindley-Milner inference (no annotations) + functional pipeline
  (range |> filter |> map |> fold)
- persistent List<T> + Map<K,V>
- the original osp_strlen (i64) / %lld width fixes preserved as the finale

Verified green via `make wasm`: compiles to wasm32, byte-matches the native
oracle under Node's WASI and the browser shim, golden suite PASS=30 FAIL=0.
Numeric recursion is intentionally omitted — checked arithmetic makes `n - 1`
/ `n * …` yield Result<int, MathError>, which won't unify with the int base
case; `fold` is the idiomatic iteration anyway. Update expectedoutput, the
tracked build/ artifacts, and the README.
Rescued from the pre-merge auto-stash: a complete benchmark run across all
22 cases and the osprey-gc backend that was uncommitted in the working tree
(the committed snapshot held only the listops case).
resolveOspreyOnPath had no caller in any suite (33/50 lines). Exercise it
directly plus resolveBuiltOsprey's PATH fallback, lifting the counted
osprey-test-env.ts toward the 95% vsix coverage gate.
…E2E coverage

The 'configured bare command resolves against a Windows PATH too' assertion
was impossible on a POSIX CI runner (path.delimiter ':' collides with Windows
drive-letter colons), and the injected-platform lldb unit branches never
registered in the instrumented host run anyway. Replace them with real
extension-host E2E: dead lldbDapPath adapter refusal, unsaved-buffer
save-before-compile, .osp mis-association self-heal, and a live ML (.ospml)
LSP sweep — covering the previously-uncovered activation/debug branches.
Authored by the vsix-coverage agent; verified tsc-clean.
@MelbourneDeveloper
MelbourneDeveloper merged commit c8a8df1 into main Jul 1, 2026
6 checks passed
@MelbourneDeveloper
MelbourneDeveloper deleted the prototyping branch July 1, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant