diff --git a/.cursor/rules/vex-project.mdc b/.cursor/rules/vex-project.mdc index bdbcc33..50d5c85 100644 --- a/.cursor/rules/vex-project.mdc +++ b/.cursor/rules/vex-project.mdc @@ -15,7 +15,7 @@ Vex is a statically typed Lisp for building MCP servers. The compiler is written ## Core Philosophy - **Pure transformations** — each compiler phase is a function: data in, data out, plus diagnostics. No global state, no singletons, no mutable statics. -- **Flat file structure** — one file per concept in `src/`. No subdirectories until a file exceeds ~500 lines. +- **One file, one concept** — each source file in `src/` owns exactly one concept. Split when a file owns two independent concerns, not by line count. - **Separate AST and HIR** — parser produces untyped `ast::*` types; type checker produces typed `hir::*` types. Never mix them. Codegen only sees HIR. - **Strict layering** — modules form a DAG. `source.rs` at the bottom, `main.rs` at the top. No dependency cycles. - **Explicit over clever** — prefer straightforward code over abstractions. No trait-based frameworks, no indirection unless it solves a concrete problem. diff --git a/.cursor/skills/add-compiler-phase/SKILL.md b/.cursor/skills/add-compiler-phase/SKILL.md index b08a705..a7c233c 100644 --- a/.cursor/skills/add-compiler-phase/SKILL.md +++ b/.cursor/skills/add-compiler-phase/SKILL.md @@ -81,4 +81,4 @@ Unit tests in `#[cfg(test)] mod tests` within the file: - No mutable statics or global state - Receive `&mut Vec` to push errors, or return `Vec` alongside output - The caller (`lib.rs`) decides whether to continue after errors -- Keep the file under ~500 lines; split only if exceeded +- Each file owns one concept; split when a file owns two independent concerns, not by line count diff --git a/.cursor/skills/pre-commit-checks/SKILL.md b/.cursor/skills/pre-commit-checks/SKILL.md index 1a80454..7e40ac0 100644 --- a/.cursor/skills/pre-commit-checks/SKILL.md +++ b/.cursor/skills/pre-commit-checks/SKILL.md @@ -20,3 +20,5 @@ cargo clippy -- -D warnings Only create the commit after both commands succeed with no errors. 3. **README trigger check** — before committing, actively verify whether any of the **update-readme** skill triggers have been reached by comparing the staged changes against the trigger conditions in that skill. If any trigger is met, run the **update-readme** skill and stage the updated `README.md` before committing. This check is mandatory on every commit, not optional. + +4. **Roadmap review** — before committing, run the **update-roadmap** skill. Compare the staged changes against the roadmap items in `docs/roadmap.md`. If any item's status changed (work started, feature completed, new gap identified), update the roadmap and stage it before committing. This check is mandatory on every commit, not optional. diff --git a/.cursor/skills/update-roadmap/SKILL.md b/.cursor/skills/update-roadmap/SKILL.md new file mode 100644 index 0000000..fec8b4e --- /dev/null +++ b/.cursor/skills/update-roadmap/SKILL.md @@ -0,0 +1,64 @@ +--- +name: update-roadmap +description: >- + Reviews and updates the project roadmap with current status of all items. + Use when committing changes, completing a feature, or when the user asks + to update the roadmap. +--- + +# Update Roadmap + +The roadmap lives at `docs/roadmap.md`. It tracks what Vex needs next, the status of each item, and links to `docs/roadmap-rationale.md` for full analysis. + +## When to Trigger + +- Before every commit (called from the pre-commit-checks skill) +- A feature or milestone is completed +- A new roadmap item is identified or started +- User explicitly asks to update the roadmap + +## Gather Current State + +Before updating, read these sources to determine what has changed: + +1. **`docs/roadmap.md`** — current roadmap with statuses +2. **`docs/roadmap-rationale.md`** — rationale and analysis for each item +3. **`src/lib.rs`** — which modules exist +4. **`src/*.rs`** — list files to see what phases and features are present +5. **Git diff / staged changes** — what changed in the current commit + +## Update Process + +1. Compare the staged changes against each roadmap item +2. If a roadmap item moved forward (new code, tests, or docs related to it), update its status: + - **Not Started** → **In Progress** when work begins on a branch + - **In Progress** → **Done** when merged to main with tests passing +3. If a completed item is not yet in the "Completed Milestones" section, move it there with the current date +4. If new work reveals a gap not on the roadmap, add it to the appropriate section +5. Cross-check `docs/roadmap-rationale.md` and `docs/language-design.md` for items, features, or gaps not yet tracked in the roadmap — add any missing items to the appropriate section with status **Not Started** +6. Update the **Last reviewed** date at the top of the document to today's date + +## What to Check + +For each roadmap section, verify: + +### Design Constraint Enforcement +- Has the compiler core / binary boundary been enforced? Check if `lib.rs` compile functions do any IO. +- Has resilient parsing been added? Check `parser.rs` for error recovery logic. + +### Type System +- Has parametric polymorphism work started? Check `types.rs` for `TypeParam`, check `typechecker.rs` for unification logic. + +### Developer Experience +- Has `vex dev`, structured logging, connected REPL, error chain traces, or test framework work started? Check `src/` for new files or CLI commands. + +### MCP Framework +- Has any MCP-specific macro or runtime work started? Check for `deftool`, `defresource`, `serve-mcp` in source or examples. + +## Rules + +- **Only report what is actually implemented.** Read the source — do not guess. +- Keep statuses consistent between the roadmap and the README phase table. +- Do not remove items from the roadmap — mark them Done and move to Completed Milestones. +- Do not add speculative items. Every item must trace back to a concrete gap identified in `docs/roadmap-rationale.md` or `docs/compiler-architecture.md`. +- **Follow `docs/documentation-guidelines.md`** — active voice, bullet points, no filler words. diff --git a/README.md b/README.md index 50e27a0..73c6d0a 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,11 @@ [![CI](https://github.com/thsfranca/vex/actions/workflows/ci.yml/badge.svg)](https://github.com/thsfranca/vex/actions/workflows/ci.yml) -A statically typed Lisp for building MCP servers. The compiler is written in Rust and transpiles to Go source code. +A statically typed Lisp for building [MCP](https://modelcontextprotocol.io/) servers. The compiler is written in Rust and transpiles to Go source code, which `go build` compiles to a native binary. **This is a study project.** -## Overview - -Vex combines S-expression syntax with compile-time type checking, targeting networked services — especially [MCP](https://modelcontextprotocol.io/) servers. The compiler pipeline produces Go source that `go build` compiles to a native binary. An alternative path runs code directly through a tree-walking interpreter for the REPL. - -## Examples +## Quick Look ```lisp (defn main [] @@ -28,62 +24,50 @@ Vex combines S-expression syntax with compile-time type checking, targeting netw ``` ```lisp -(deftype Point (x Float) (y Float)) - -(defn distance [p : Point] : String - (str "(" (. p x) ", " (. p y) ")")) +(defmacro unless [test body] + (list (quote if) test (quote nil) body)) (defn main [] - (let [origin (Point 0.0 0.0) - target (Point 3.0 4.0)] - (println (str "origin = " (distance origin))) - (println (str "target = " (distance target))))) + (unless (> 1 10) + (println "1 is not greater than 10"))) ``` -## Usage +More examples in [`examples/`](examples/). + +## Features + +- **Type system** — `Int`, `Float`, `String`, `Bool`, `Option`, `Result`, records (`deftype`), unions (`defunion`) +- **Functions** — `defn`, `def`, `fn` (lambdas), higher-order functions, closures +- **Control flow** — `if`, `cond`, `match` (pattern matching), `and`, `or`, `let` +- **Macros** — `defmacro` with `quote`/`unquote`/`splice`, automatic hygiene, self-hosted prelude +- **Collections** — `List`, `Map`, `each`, `range`, `map`, `filter` +- **Modules** — `module`, `export`, `import`, Go interop (`import-go`) +- **Concurrency** — `spawn`, `channel`, `send`, `recv`, `select` +- **REPL** — interactive `vex repl` with multi-line input and persistent state + +## Getting Started + +### Prerequisites + +- [Rust](https://www.rust-lang.org/tools/install) (stable) +- [Go](https://go.dev/dl/) (1.21+) + +### Build the compiler + +```bash +cargo build --release +``` + +### Compile and run a program ```bash vex build hello.vx # Compile to ./hello binary vex build hello.vx -o server # Custom output name -vex build hello.vx --emit-go . # Also write generated Go source vex run hello.vx # Build and run immediately vex repl # Interactive REPL +vex build hello.vx --emit-go . # Write generated Go source for inspection ``` -## Current Status - -All compiler phases are implemented and pass 551 tests. The self-hosted macro system (`defmacro`) supports user-defined compile-time macros with automatic hygiene. - -### Compiler Phases - -| Phase | Status | -|-------|--------| -| `source.rs` — FileId, Span, SourceMap | Done | -| `diagnostics.rs` — Diagnostic, Severity, formatting | Done | -| `lexer.rs` — Tokenizer | Done | -| `ast.rs` — Untyped AST types | Done | -| `parser.rs` — Recursive descent parser | Done | -| `types.rs` / `hir.rs` / `builtins.rs` — Type system | Done | -| `macro_expand.rs` — Compiler macros + self-hosted `defmacro` with hygiene | Done | -| `typechecker.rs` — AST → HIR | Done | -| `codegen.rs` — HIR → Go source | Done | -| `interpreter.rs` — HIR → Value (tree-walking eval) | Done | -| `lib.rs` / `main.rs` — Full pipeline, CLI | Done | - -### Language Features - -- **Primitives:** integers, floats, strings, booleans, nil -- **Functions:** `defn`, `def`, `fn` (lambdas), higher-order functions -- **Control flow:** `if`, `cond`, `and`, `or`, `let`, pattern matching (`match`) -- **Macros:** `defmacro`, `quote`, `unquote`, `splice`, macro helpers, automatic hygiene via `gensym` -- **Data types:** records (`deftype`), field access (`.`), unions (`defunion`) -- **Built-in types:** `Option`, `Result`, `List`, `Map` -- **Collections:** `each`, `range`, `map`, `filter` -- **Modules:** `module`, `export`, `import`, Go interop (`import-go`) -- **Concurrency:** `spawn`, `channel`, `send`, `recv` -- **REPL:** interactive `vex repl` with multi-line input and persistent state -- **Built-in functions:** `println`, `str`, `mod`, arithmetic and comparison operators - ## Architecture ``` @@ -91,18 +75,47 @@ Source → Lexer → Parser → Macro Expand → Type Checker → Codegen → go → Interpreter (REPL) ``` +Each compiler phase is a pure function: data in, data out, plus diagnostics. No global state, no singletons. + +| Phase | File | Description | +|-------|------|-------------| +| Source tracking | `source.rs` | `FileId`, `Span`, `SourceMap` | +| Diagnostics | `diagnostics.rs` | Error/warning accumulation and formatting | +| Lexer | `lexer.rs` | Source text → token stream | +| AST | `ast.rs` | Untyped syntax tree types | +| Parser | `parser.rs` | Recursive descent, tokens → AST | +| Macro expansion | `macro_expand.rs` | AST → AST, prelude + `defmacro` with hygiene | +| Type system | `types.rs`, `hir.rs`, `builtins.rs` | Semantic types, typed HIR, built-in registry | +| Type checker | `typechecker.rs` | AST → HIR with type inference | +| Code generation | `codegen.rs` | HIR → Go source | +| Interpreter | `interpreter.rs` | HIR → Value (tree-walking, for REPL) | +| Pipeline | `lib.rs`, `main.rs` | Full compiler pipeline and CLI | + +## Current Status + +All compiler phases are implemented and pass 551 tests. The pipeline compiles Vex source to working Go binaries end-to-end. The self-hosted macro system (`defmacro`) supports user-defined compile-time macros with automatic hygiene. + +See [`docs/roadmap.md`](docs/roadmap.md) for planned features: parametric polymorphism, error propagation macros, exhaustiveness checking, structured concurrency, formatter, LSP, and the MCP framework. + ## Documentation -- [`docs/language-design.md`](docs/language-design.md) — syntax, type system, grammar, backend strategy, design decisions -- [`docs/compiler-architecture.md`](docs/compiler-architecture.md) — pipeline, file structure, phase contracts, testing strategy -- [`docs/dependency-management.md`](docs/dependency-management.md) — `vex.mod` manifest, `vex get`, global cache, Go module integration -- [`docs/mvp.md`](docs/mvp.md) — MVP definition and success criteria +| Document | Contents | +|----------|----------| +| [`language-design.md`](docs/language-design.md) | Syntax, type system, grammar, backend strategy, design decisions | +| [`compiler-architecture.md`](docs/compiler-architecture.md) | Pipeline, file structure, phase contracts, testing strategy | +| [`dependency-management.md`](docs/dependency-management.md) | `vex.mod` manifest, `vex get`, global cache, Go module integration | +| [`roadmap.md`](docs/roadmap.md) | Feature roadmap with priorities and status | +| [`roadmap-rationale.md`](docs/roadmap-rationale.md) | Design analysis and trade-offs behind roadmap items | +| [`installation.md`](docs/installation.md) | Installation process, release pipeline, distribution channels | +| [`mvp.md`](docs/mvp.md) | MVP definition and success criteria | ## Development ```bash -cargo build -cargo test +cargo build # Build the compiler +cargo test # Run all 551 tests +cargo clippy # Lint +cargo fmt # Format ``` ## License diff --git a/docs/compiler-architecture.md b/docs/compiler-architecture.md index 1d8d436..efbd299 100644 --- a/docs/compiler-architecture.md +++ b/docs/compiler-architecture.md @@ -50,6 +50,11 @@ Source (.vx) └────────┬────────┘ │ ▼ +┌───────────────────────┐ +│ Summary Extraction │ Vec → ModuleSummary (future, see roadmap §9) +└───────────┬───────────┘ + │ + ▼ ┌──────────────┐ │ Type Checker │ &[ast::TopForm] → hir::Module └──────┬───────┘ @@ -86,7 +91,7 @@ Macro expansion sits between Parser and Type Checker: ## 2. File Structure -Flat files. No directories for modules until a file exceeds ~500 lines. Each file owns one concept. +Each file owns one concept. Split when a file owns two independent concerns that don't share state — not by line count. See `docs/roadmap-rationale.md` §0 for the rationale behind this constraint. ``` src/ @@ -562,7 +567,7 @@ func (McpMessage_Notification) isMcpMessage() {} ``` - The Vex type checker validates all types at the Vex level -- The Go interface is not generic — it uses `any` where Vex type parameters appear — because Go's generics cannot express the full Vex type system +- The Go interface uses `any` where Vex type parameters appear in union variants. Go 1.26 (2026) relaxed recursive type parameter constraints, reducing the impedance mismatch for many patterns — but some Vex type combinations still fall back to `any` - This is acceptable since the generated code is correct by construction after type checking ### Concurrency Primitives @@ -682,12 +687,22 @@ These are tools the Vex compiler and toolchain provide, independent of any frame | P0 | Error diagnostics | Compiler (`diagnostics.rs`) | Span-based errors with source snippets, line numbers, and underlines. The single most important DX feature for a new language. | | P0 | REPL | Interpreter (`interpreter.rs`) | Tree-walking evaluation of typed HIR. Instant feedback, no compile cycle. The primary development workflow for a Lisp. | | P0 | `--emit-go` | CLI (`main.rs`) | Write the generated Go module to a directory instead of deleting it. Essential for understanding and debugging codegen output. | +| P1 | `vex fmt` (formatter) | CLI + `formatter.rs` | Opinionated code formatter for `.vx` files. Table stakes for modern languages. See `roadmap-rationale.md` §6. | | P1 | `vex dev` (hot reload) | CLI | File watcher that recompiles and restarts on source changes. Sub-second reload is more productive than any debugger for server development. | | P1 | Structured logging | Stdlib (`vex.log`) | Key-value structured log output. The server developer's primary diagnostic tool in production and development. | +| P1 | Source location mapping | Codegen (`codegen.rs`) | Emit `//line` directives in generated Go so stack traces and profiling tools point to `.vx` source. See `roadmap-rationale.md` §7. | +| P1 | Go toolchain detection | CLI (`main.rs`) | Validate Go installation and version before compilation with clear error messages. See `roadmap-rationale.md` §8. | | P2 | Connected REPL | Toolchain | REPL that connects to a running `vex dev` process (nREPL model). Evaluate expressions in the server's context, redefine functions without restarting, inspect live state. | | P2 | Error chain traces | Runtime / stdlib | When a `Result` error propagates through multiple functions, display the full chain with source locations. Makes error flows debuggable without a traditional stack trace debugger. | | P3 | Test framework | Stdlib (`vex.test`) | Assertions, test discovery, test runner. General-purpose, not framework-specific. | +### Editor tooling + +| Priority | Tool | Rationale | +|----------|------|-----------| +| P1 | Tree-sitter grammar | Syntax highlighting across Neovim, Helix, Zed, Emacs, VS Code. Trivial for s-expression syntax. See `roadmap-rationale.md` §11. | +| P2 | `vex lsp` (LSP) | Language server as a CLI subcommand. Map-reduce architecture enabled by per-file independence. See `roadmap-rationale.md` §10. | + ### What belongs in the MCP framework, not the language The following serve MCP server authors specifically and will be part of the future MCP framework (`vex.mcp`), built as a library on top of the language-level tools above: @@ -706,7 +721,6 @@ The following serve MCP server authors specifically and will be part of the futu |---------|-----------| | **String interning** | Use `String` everywhere. Optimize later if profiling shows it matters. | | **Arena allocation** | Use `Box` and `Vec`. Swap for arenas later if needed. | -| **User-defined macros (`defmacro`)** | Implemented. All macros — including core control flow (`cond`, `and`, `or`) defined in the self-hosted prelude — execute via a dedicated AST evaluator with automatic hygiene. See `language-design.md` §4.5 and §14.11. | | **Multi-file compilation** | `SourceMap` supports `FileId` from day one, but the pipeline processes one file at a time. | -| **Error recovery in parser** | Stop at first error initially. Accumulate multiple errors later. | -| **LSP / incremental compilation** | Not a concern at this stage. | +| **Error recovery in parser** | Planned — see `roadmap.md` "Resilient parsing". Stop at first error for now. | +| **LSP / incremental compilation** | Planned — see `roadmap-rationale.md` §9 (summary extraction) and §10 (LSP). Not needed until multi-file support exists. | diff --git a/docs/dependency-management.md b/docs/dependency-management.md index 6bef9fd..f101b4c 100644 --- a/docs/dependency-management.md +++ b/docs/dependency-management.md @@ -278,11 +278,12 @@ When `vex build` runs, the compiler resolves dependencies in this order: 1. **Read `vex.mod`** — parse the manifest to find all `require`, `go`, and `replace` entries 2. **Apply replacements** — for each `replace` directive, use the local path instead of the cache 3. **Locate Vex dependencies** — find each required Vex package in the global cache. If missing, error with a message suggesting `vex deps` or `vex get` -4. **Compile Vex dependencies** — compile each dependency module before the main module (existing `compile_single` flow in `lib.rs`) -5. **Collect Go dependencies** — merge the `go` section from `vex.mod` with any Go dependencies declared by Vex package dependencies (transitive) -6. **Generate `go.mod`** — produce a `go.mod` with all Go `require` directives -7. **Run `go mod download`** — download Go dependencies before `go build` -8. **Build** — run `go build` as usual +4. **Extract summaries** — extract type signatures and exports from each dependency (future: summary extraction phase, see `roadmap-rationale.md` §9). Until summary extraction is implemented, the compiler compiles each dependency fully +5. **Compile Vex dependencies** — compile each dependency module before the main module (existing `compile_single` flow in `lib.rs`) +6. **Collect Go dependencies** — merge the `go` section from `vex.mod` with any Go dependencies declared by Vex package dependencies (transitive) +7. **Generate `go.mod`** — produce a `go.mod` with all Go `require` directives +8. **Run `go mod download`** — download Go dependencies before `go build` +9. **Build** — run `go build` as usual ### Pipeline Diagram @@ -476,3 +477,4 @@ This means every existing Vex program compiles without changes. | Private repository authentication | Relies on Git credentials (SSH keys, credential helpers) already configured on the machine | | Workspaces / multi-module projects | Can be addressed with `replace` directives for now | | Dependency graph visualization | Nice-to-have, not needed for v0.1.0 | +| `vex tidy` (remove unused deps) | Can be added once dependency resolution and import tracking are stable | diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..77c1177 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,297 @@ +# Installation + +This document describes how users install Vex on their system, how the release pipeline produces binaries, and the planned distribution channels. + +For platform targets, see `language-design.md` §12. For the Go toolchain requirement, see `roadmap-rationale.md` §8. + +--- + +## 1. Quick Install + +A single command installs Vex on macOS and Linux: + +```bash +curl -fsSL https://raw.githubusercontent.com/thsfranca/vex/main/install.sh | sh +``` + +The installer: + +1. Detects the operating system and CPU architecture +2. Downloads the correct binary from GitHub Releases +3. Installs to `~/.vex/bin/vex` +4. Adds `~/.vex/bin` to the shell's `PATH` (auto-detects bash, zsh, and fish) +5. Checks for a Go toolchain (>= 1.21) and prints install instructions if missing + +After installation, restart the shell or source the rc file: + +```bash +source ~/.zshrc # or ~/.bashrc, depending on your shell +vex version # verify it works +``` + +### Installer Options + +- **`--no-modify-path`** — skip automatic PATH modification. The installer prints the export line instead of writing it +- **`VEX_VERSION`** — install a specific version instead of the latest: + +```bash +VEX_VERSION=v0.2.0 curl -fsSL https://raw.githubusercontent.com/thsfranca/vex/main/install.sh | sh +``` + +--- + +## 2. Manual Installation + +Download a release archive from [GitHub Releases](https://github.com/thsfranca/vex/releases), extract the binary, and place it somewhere on your `PATH`: + +```bash +tar -xzf vex-v0.1.0-darwin-arm64.tar.gz +mv vex ~/.local/bin/ # or /usr/local/bin/, or any directory on PATH +``` + +### Windows + +Download the `.zip` archive from GitHub Releases, extract `vex.exe`, and add its directory to the system `PATH` via Settings > System > Environment Variables. + +--- + +## 3. Prerequisites + +Vex compiles to Go source code and requires the Go toolchain to produce binaries. + +- **Minimum Go version:** 1.21 +- The `vex build` and `vex run` commands check for Go automatically and produce clear error messages if Go is missing or outdated (see `roadmap-rationale.md` §8) + +If Go is not installed: + +``` +error: Go toolchain not found + +Vex compiles to Go source code and requires the Go toolchain to produce binaries. + +Install Go: https://go.dev/dl/ + macOS: brew install go + Linux: sudo apt install golang (or download from go.dev) + Windows: winget install GoLang.Go +``` + +--- + +## 4. Install Directory + +The installer places the binary at `~/.vex/bin/vex`. This directory also serves as the root for other Vex-managed data: + +``` +~/.vex/ + bin/ + vex # the compiler binary + cache/ # global dependency cache (see dependency-management.md §6) +``` + +The `VEX_HOME` environment variable overrides the default `~/.vex/` location. + +--- + +## 5. PATH Setup + +The installer automatically adds `~/.vex/bin` to the user's shell configuration. The behavior follows the same pattern as rustup, Bun, and other modern toolchain installers: + +- **bash** — appends to `~/.bash_profile` (falls back to `~/.bashrc` if `~/.bash_profile` does not exist) +- **zsh** — appends to `~/.zshrc` +- **fish** — appends to `~/.config/fish/config.fish` + +The line the installer adds: + +```bash +# bash / zsh +export PATH="$HOME/.vex/bin:$PATH" +``` + +```fish +# fish +set -gx PATH "$HOME/.vex/bin" $PATH +``` + +### Idempotency + +The installer checks if the PATH line already exists before writing. Running the installer multiple times does not produce duplicate entries. The check uses a simple grep for `.vex/bin` in the target file. + +### Opt-out + +Pass `--no-modify-path` to skip automatic PATH modification: + +```bash +curl -fsSL https://raw.githubusercontent.com/thsfranca/vex/main/install.sh | sh -s -- --no-modify-path +``` + +The installer prints the export line so the user can add it manually. + +--- + +## 6. Release Artifacts + +Each release produces pre-built binaries for all primary targets: + +| Artifact name | OS | Architecture | Format | +| ---------------------------------- | ------- | ------------ | -------- | +| `vex-v{version}-darwin-amd64.tar.gz` | macOS | x86_64 | tar.gz | +| `vex-v{version}-darwin-arm64.tar.gz` | macOS | ARM64 | tar.gz | +| `vex-v{version}-linux-amd64.tar.gz` | Linux | x86_64 | tar.gz | +| `vex-v{version}-linux-arm64.tar.gz` | Linux | ARM64 | tar.gz | +| `vex-v{version}-windows-amd64.zip` | Windows | x86_64 | zip | + +Naming follows the `{name}-v{version}-{os}-{arch}.{ext}` convention, using Go's `GOOS`/`GOARCH` naming for familiarity with the target audience. + +Each archive contains a single `vex` binary (or `vex.exe` on Windows). No wrapper scripts, no configuration files, no runtime dependencies. + +### Linux Static Linking + +Linux binaries are compiled against `musl` (not glibc) to produce fully static executables. This eliminates glibc version mismatches across distributions — the binary runs on any Linux system regardless of the installed C library. + +--- + +## 7. Release Pipeline + +A GitHub Actions workflow builds and publishes release artifacts when a version tag is pushed: + +``` +git tag v0.1.0 +git push origin v0.1.0 +``` + +### Build Matrix + +| Rust target | Runner | Notes | +| -------------------------------- | -------------- | ---------------------------------- | +| `x86_64-apple-darwin` | `macos-13` | Intel Mac | +| `aarch64-apple-darwin` | `macos-latest` | Apple Silicon (M-series) | +| `x86_64-unknown-linux-musl` | `ubuntu-latest`| Static binary, musl libc | +| `aarch64-unknown-linux-musl` | `ubuntu-latest`| Cross-compiled via `cross` | +| `x86_64-pc-windows-msvc` | `windows-latest`| MSVC toolchain | + +### Workflow Steps + +1. **Build** — each matrix entry compiles with `cargo build --release --target`, then packages the binary into the archive format (tar.gz for Unix, zip for Windows) +2. **Release** — after all builds succeed, creates a GitHub Release with all artifacts attached and auto-generated release notes + +--- + +## 8. CLI Version and Help + +The Vex binary supports standard version and help flags: + +```bash +vex version # prints: vex 0.1.0 +vex --version # same +vex -V # same + +vex --help # prints usage with all subcommands +vex -h # same +``` + +The version string uses the version from `Cargo.toml` via `env!("CARGO_PKG_VERSION")`, ensuring the binary version always matches the release tag. + +--- + +## 9. Go Toolchain Detection + +Before compiling any `.vx` file, `vex build` and `vex run` validate the Go toolchain: + +1. Check if `go` exists on `PATH` by running `go version` +2. If `go` is not found, print the "Go toolchain not found" error (§3) and exit +3. If `go` is found, parse the version from the output (format: `go version go1.X.Y ...`) +4. If the version is below 1.21, print: + +``` +error: Go 1.21 or later required (found go1.18.3) + +Update Go: https://go.dev/dl/ +``` + +This check runs once per build invocation. It adds one subprocess call (~10ms) before compilation. + +The check lives in the CLI (`main.rs`), not the compiler core (`lib.rs`). The compiler core remains a pure function with no IO. + +--- + +## 10. Future Distribution Channels + +These channels are planned but not implemented yet: + +### Homebrew (macOS) + +A Homebrew tap (`homebrew-vex`) with a formula that: + +- Downloads the correct binary from GitHub Releases +- Links it into the Homebrew prefix +- Declares Go as a dependency +- Validates Go version on install + +```bash +brew tap thsfranca/vex +brew install vex +``` + +### APT / RPM (Linux) + +Debian and RPM packages for distribution via: + +- A PPA (Ubuntu/Debian) +- A Copr repository (Fedora/RHEL) + +Packages declare `golang` as a runtime dependency. + +### Scoop / WinGet (Windows) + +- **Scoop** — a manifest in a Scoop bucket repository +- **WinGet** — a manifest in the WinGet Community Repository + +### Shell Completions + +Generate shell completion scripts for bash, zsh, and fish via a CLI subcommand: + +```bash +vex completions bash > /etc/bash_completion.d/vex +vex completions zsh > ~/.zfunc/_vex +vex completions fish > ~/.config/fish/completions/vex.fish +``` + +The installer can optionally install completions automatically alongside the binary. + +--- + +## 11. Design Decisions + +### 1. `~/.vex/bin/` as the install directory — not `/usr/local/bin/` + +- No `sudo` required for installation +- Matches rustup (`~/.cargo/bin/`), Deno (`~/.deno/bin/`), and Bun (`~/.bun/bin/`) +- Co-locates with the global cache (`~/.vex/cache/`) under a single root +- Users who prefer a system-wide install can move the binary manually + +### 2. Automatic PATH modification — opt-out, not opt-in + +- rustup and Bun modify PATH automatically — the established pattern for developer toolchains +- Deno does not auto-modify PATH and faces persistent usability complaints (GitHub issue #286, PR #295) +- Auto-setup reduces first-run friction to zero for new users +- Experienced users who manage PATH manually use `--no-modify-path` +- Idempotent writes prevent accumulating duplicate entries + +### 3. Go as an explicit dependency — not bundled + +- Bundling Go adds ~150MB to the distribution +- Zig bundles its C compiler because no standard C compiler exists across platforms. Go has a single canonical installer at go.dev +- Gleam detects Erlang and prints install instructions — the same approach works for Vex +- Clear error messages (§9) make the dependency visible at the right moment + +### 4. musl for Linux — not glibc + +- glibc version mismatches cause "GLIBC_2.XX not found" errors on older distributions +- musl produces fully static binaries that run on any Linux kernel version +- Performance difference is negligible for a compiler binary (startup-dominated, not compute-bound) + +### 5. Artifact naming uses Go-style `os-arch` — not Rust triple + +- Vex targets Go developers. `darwin-arm64` is familiar; `aarch64-apple-darwin` is not +- Shorter names reduce visual clutter in release pages +- The installer maps `uname` output to these names, not Rust targets diff --git a/docs/language-design.md b/docs/language-design.md index 9814173..395ffb4 100644 --- a/docs/language-design.md +++ b/docs/language-design.md @@ -24,6 +24,8 @@ 12. [Platform Support](#12-platform-support) 13. [Example Programs](#13-example-programs) 14. [Design Decisions](#14-design-decisions) +- [Appendix A: Prior Art](#appendix-a-prior-art) +- [Appendix B: Glossary](#appendix-b-glossary) --- @@ -49,7 +51,7 @@ Vex is a statically typed, Lisp-based language for building networked services ## 2. Design Principles 1. **Explicitness over magic** — types are inferred where unambiguous but always expressible; no hidden coercions -2. **Composition over inheritance** — algebraic data types and traits, not class hierarchies +2. **Composition over inheritance** — algebraic data types and higher-order functions, not class hierarchies 3. **Errors are values** — Result types instead of exceptions; errors must be handled or explicitly propagated 4. **Concurrency is structural** — lightweight tasks and channels built into the language, not a library bolt-on 5. **Interop is practical** — direct Go package imports for ecosystem access; JSON and HTTP in the standard library @@ -279,13 +281,16 @@ For example, the `or` macro in the prelude introduces a temporary binding. With ## 5. Type System -### 5.1 Approach: Hindley-Milner with Extensions +### 5.1 Approach: Explicit Types with Local Inference -Based on Hindley-Milner type inference (ML, Haskell, Typed Racket), extended with: +Function signatures require explicit type annotations on parameters. The compiler infers types locally within function bodies — `let` bindings infer from their initializer, return types infer from the body when the annotation is omitted. No global inference across function boundaries. -- **Algebraic Data Types** — sum and product types -- **Parametric polymorphism** — generics -- **Traits / Protocols** — ad-hoc polymorphism +This follows Rust and Go's direction: the programmer states the contract (function signature), and the compiler handles the bookkeeping inside the body. Hindley-Milner was considered and rejected — full HM infers principal types globally, which conflicts with "explicit over clever" (§2.1): type errors surface at unification points far from the actual mistake, and code becomes unreadable without LSP tooling that Vex does not have. + +The type system includes: + +- **Algebraic Data Types** — sum types (`defunion`) and product types (`deftype`) +- **Parametric polymorphism** — explicit type variables in function signatures (planned; see roadmap §1) - **Result types** — error handling via `(Result T E)` - **Option types** — nullable values via `(Option T)` @@ -313,27 +318,12 @@ Based on Hindley-Milner type inference (ML, Haskell, Typed Racket), extended wit | Record (product type) | `(deftype Name (field1 T1) ...)` | | Union (sum type) | `(defunion Name (Variant1 T1) ...)` | -### 5.4 Traits / Protocols - -``` -(deftrait Serializable - (serialize [self] : String) - (deserialize [s : String] : (Result Self Error))) - -(impl Serializable for ToolInput - (defn serialize [self] (json.encode self)) - (defn deserialize [s] (json.decode s))) -``` +### 5.4 Type Inference -### 5.5 Type Inference - -- Types are inferred in most local contexts -- Return types are inferred from the function body when omitted (e.g., a body ending in `println` infers to `Unit`) -- Return type annotations are only needed when the compiler can't infer unambiguously +- `let` bindings infer their type from the initializer expression +- Return types infer from the function body when the annotation is omitted (e.g., a body ending in `println` infers to `Unit`) - Explicit annotations are **required** on: - - Top-level function parameter types - - Trait implementations - - Ambiguous expressions + - All function parameter types (`defn` and `fn`/lambda) --- @@ -600,8 +590,6 @@ top-form = module-decl | defn-form | deftype-form | defunion-form - | deftrait-form - | impl-form | defmacro-form | expression ; @@ -621,10 +609,6 @@ deftype-form = "(" "deftype" SYMBOL { field-decl } ")" ; defunion-form = "(" "defunion" SYMBOL { variant-decl } ")" ; -deftrait-form = "(" "deftrait" SYMBOL { trait-method } ")" ; - -impl-form = "(" "impl" SYMBOL "for" SYMBOL { defn-form } ")" ; - defmacro-form = "(" "defmacro" SYMBOL param-list body ")" ; param-list = "[" { param } "]" ; @@ -635,8 +619,6 @@ field-decl = "(" SYMBOL type ")" ; variant-decl = "(" SYMBOL { type } ")" ; -trait-method = "(" SYMBOL param-list ":" type ")" ; - type-ann = ":" type ; type = SYMBOL @@ -776,9 +758,9 @@ The Vex compiler is written in Rust. It outputs Go source code, which the Go too #### Known limitations -- Go's generics may not fully express Vex's type system in generated code +- Go 1.26 (2026) relaxed recursive type parameter constraints, reducing the generics impedance mismatch — but some Vex type combinations still require `any` in generated code - GC pauses are small but not zero -- Transpiled code is harder to debug than natively compiled code +- Transpiled code is harder to debug than natively compiled code — `//line` directives (see `roadmap-rationale.md` §7) mitigate this #### Why Go is the only backend @@ -835,6 +817,18 @@ Source (.vx) Lightweight tasks and channels, inspired by Go's goroutines and CSP. +### Primitives + +| Vex | Go | Description | +|-----|-----|-------------| +| `(spawn expr)` | `go func() { expr }()` | Fire-and-forget goroutine | +| `(channel T)` | `make(chan T)` | Unbuffered channel | +| `(channel T size)` | `make(chan T, size)` | Buffered channel | +| `(send ch val)` | `ch <- val` | Send a value to a channel | +| `(recv ch)` | `<-ch` | Receive a value from a channel | + +### Example + ``` (let [ch (channel Int 10)] (spawn @@ -844,10 +838,22 @@ Lightweight tasks and channels, inspired by Go's goroutines and CSP. (fn [_] (println (recv ch))))) ``` -- `spawn` → goroutine -- `channel` → Go channel +### `select` + +`select` multiplexes across multiple channel operations. The runtime picks whichever operation is ready first. If none is ready and a `:default` clause exists, the default executes immediately. Without `:default`, `select` blocks. + +``` +(select + [(recv ch1) msg (println (str "got: " msg))] + [(send ch2 value) (println "sent")] + [:default (println "nothing ready")]) +``` + +Each clause is a bracketed triple: a channel operation, a binding (for `recv`) or nothing (for `send`), and a body. The generated Go is a direct `select { ... }` statement. + +### Structured concurrency (planned) -This gives Vex a battle-tested concurrent runtime for free. +`spawn` is fire-and-forget — no mechanism to wait for completion, collect results, or cancel on failure. `task-group` (see `roadmap-rationale.md` §5) adds scoped concurrency: all tasks in a group must complete before the group exits, and errors cancel remaining tasks. --- @@ -890,12 +896,15 @@ All targets are supported via Go's `GOOS`/`GOARCH` cross-compilation from a sing ### Compiler Distribution -Distributed as a native Rust binary per platform: +Distributed as a native Rust binary per platform. The primary installation method is a `curl | sh` installer script that downloads the correct binary, installs to `~/.vex/bin/`, and auto-configures PATH. -- GitHub Releases -- Homebrew (macOS) -- APT/RPM (Linux) -- Scoop or WinGet (Windows) +Distribution channels (see `docs/installation.md` for the full design): + +- **Installer script** — `curl -fsSL https://raw.githubusercontent.com/thsfranca/vex/main/install.sh | sh` +- **GitHub Releases** — pre-built archives for all primary targets +- **Homebrew** (macOS) — planned +- **APT/RPM** (Linux) — planned +- **Scoop or WinGet** (Windows) — planned --- @@ -1065,6 +1074,71 @@ Distributed as a native Rust binary per platform: - `Syntax` is a built-in union type (like `Option` and `Result`) representing Vex syntax as data - Macros are erased after expansion — `defmacro` forms do not appear in the HIR or generated Go output +### 12. Mutability — rebinding only, no mutation + +- All `let` bindings are reassignable with `set!`: + +``` +(let [x 1] + (set! x 2) + x) ;; => 2 +``` + +- No separate `let mut` form — every binding is reassignable +- Compound data (records, lists, maps) is persistent: `set!` rebinds the name, it does not mutate the structure +- This trades per-binding mutation control for simplicity and aligns with Clojure's value semantics +- Go variables are mutable, so `set!` compiles to a plain reassignment + +### 13. Closure capture — by value + +- Closures capture bindings by value (Go's default for non-pointer types) +- Closures see the value of a binding at the time of capture, not at the time of call +- No aliasing hazards between a closure and its enclosing scope after capture +- Hygienic macros also prevent accidental capture: macro-introduced bindings do not leak into the expansion site unless explicitly spliced + +### 14. Error type — message-only + +- `Error` is a built-in opaque type that wraps a message string +- It is the error branch of `Result`: + +``` +(deftype Result [T] + (Ok T) + (Err Error)) +``` + +- `Error` carries only a message — no structured fields, no error codes, no stack trace +- MCP servers map errors to JSON-RPC error responses (code + message pair) +- Richer structured errors (error chains, source locations) are planned as DX improvements (see `compiler-architecture.md` §13) + +### 15. JsonValue — dynamic JSON at protocol boundaries + +`JsonValue` is the dynamic JSON type used at MCP protocol boundaries: + +| Variant | Meaning | +|---------|---------| +| `JsonNull` | JSON `null` | +| `JsonBool Bool` | JSON boolean | +| `JsonNumber Float` | JSON number | +| `JsonString String` | JSON string | +| `JsonArray (List JsonValue)` | JSON array | +| `JsonObject (Map String JsonValue)` | JSON object | + +- `JsonValue` exists because MCP tool inputs and outputs are arbitrary JSON +- Typed functions convert to/from `JsonValue` at protocol boundaries; internal code uses concrete types + +### 16. Runtime failure behavior — panics for programmer errors + +Some operations fail at runtime despite passing type checking: + +- `(/ x 0)` — division by zero panics (Go `panic`) +- `(recv ch)` on a closed channel — panics +- `(get m key)` on a missing key — returns the zero value (Go map semantics) + +- Vex does not recover from panics — a panic in a `spawn`-ed task crashes the process +- MCP servers are short-lived request handlers, and an unrecoverable error should surface immediately +- Structured error handling (`Result`/`match`) covers all expected failure modes; panics signal programmer errors + --- ## Appendix A: Prior Art @@ -1077,6 +1151,8 @@ Distributed as a native Rust binary per platform: | **Hy** | Lisp transpiled to Python | | **Fennel** | Lisp transpiled to Lua | | **Elm** | ML-family, Result types, no runtime exceptions | +| **Gleam** | Statically typed, functional, compiles to Erlang/JS; exhaustive pattern matching, `Result`-based errors, no exceptions | +| **Coalton** | Statically typed Lisp on Common Lisp; explicit type annotations, fixed-arity functions (dropped currying in 0.2), typeclass polymorphism | | **Go** | Target language, concurrency model inspiration | ## Appendix B: Glossary @@ -1086,7 +1162,7 @@ Distributed as a native Rust binary per platform: | **S-expression** | Symbolic expression — nested list notation `(op arg1 arg2)` | | **Homoiconicity** | Code-as-data; the program's syntax tree is a data structure in the language | | **ADT** | Algebraic Data Type — sum types (tagged unions) + product types (records) | -| **HM** | Hindley-Milner — a type inference algorithm that can infer types without annotations | +| **HM** | Hindley-Milner — a type inference algorithm that infers types without annotations; Vex does not use HM (see §5.1) | | **MCP** | Model Context Protocol — open protocol for LLM-tool integration | | **JSON-RPC** | Remote procedure call protocol encoded in JSON | | **TCO** | Tail Call Optimization — converting recursive tail calls into loops | diff --git a/docs/mvp.md b/docs/mvp.md index 58dce2a..0a6a40c 100644 --- a/docs/mvp.md +++ b/docs/mvp.md @@ -1,6 +1,6 @@ # Vex MVP Definition -**Status:** Active +**Status:** Complete **Date:** 2026-03-21 --- diff --git a/docs/roadmap-rationale.md b/docs/roadmap-rationale.md new file mode 100644 index 0000000..4ce95c0 --- /dev/null +++ b/docs/roadmap-rationale.md @@ -0,0 +1,864 @@ +# Roadmap Rationale + +This document records where Vex stands relative to the state of the art, what gaps exist, why each gap matters, and what trade-offs each solution involves. It guides the project's evolution without prescribing a fixed timeline. + +Every recommendation below must pass through Vex's design constraints (see §0). + +--- + +## 0. Design Constraints + +These constraints filter which techniques and architectures apply to Vex. Each constraint is grounded in a concrete reason — not preference, not convention. + +### Constraints carried forward + +**Explicit over clever.** Prefer clarity over abstraction. Type annotations on function signatures, not global inference. This aligns with the direction of Rust, Go, and Zig — the field trends toward explicitness, not away from it. + +**Separate AST and HIR.** The parser produces untyped `ast::*` types. The type checker produces typed `hir::*` types. Codegen only sees HIR. Standard practice in every serious modern compiler (Rust, Swift, Zig). + +**Strict layering.** Modules form a DAG. No dependency cycles. Universal best practice. + +**No indirection unless it solves a concrete problem.** No trait-based frameworks, no visitor patterns at this scale. Exhaustive matching on enums is the correct approach — Rust's compiler catches missing arms when a new variant is added. A visitor pattern adds indirection to solve a problem that `cargo build` already solves for free. This holds as long as Vex stays at its current scale (~15 type variants, a handful of consumers). Revisit if the project reaches rustc-level complexity (dozens of passes, hundreds of variants). + +### Constraints revised + +**Per-file independence** (replaces "pure transformations"). Each `.vx` file can be parsed, macro-expanded, and summarized (exported types and signatures extracted) without reading any other file. Cross-file analysis uses only summaries, not full ASTs. + +Why this replaces "pure transformations": the old constraint described how to write the compiler (functions in, data out). The new constraint describes how to *design the language* so the compiler stays simple. Vex's module system (`module`, `export`, `import`) already provides explicit boundaries. There are no cross-file macros, no glob imports, no open trait impls. These properties enable the **map-reduce IDE architecture** (used by IntelliJ and Sorbet) — the simplest of the three architectures described in matklad's "Three Architectures for a Responsive IDE." Future language features must not break per-file independence, or Vex will be forced into a query-based architecture (Salsa, rust-analyzer) with far higher complexity. + +The batch pipeline (`vex build`) stays pure — data in, data out, discard everything. Per-file independence is a stronger constraint that also governs a future LSP: index each file independently, merge indexes, resolve lazily, blow away caches on change. + +**One file, one concept** (replaces "flat file structure"). A file owns one concept. Split when a file owns two independent concerns that don't share state. No line count threshold. + +Why the 500-line threshold was wrong: `typechecker.rs` (3,560 lines), `codegen.rs` (3,060 lines), and `parser.rs` (2,437 lines) are each a single struct with methods that all operate on the same state. Splitting them into multiple files adds `pub` annotations, `mod` declarations, and cross-file navigation without changing the dependency structure. Zig's `Sema.zig` (the semantic analyzer) is over 20,000 lines in a single file by deliberate choice. Gleam splits by crate (`compiler-core`, `compiler-cli`, `language-server`), not by line count within a crate. + +The meaningful split for Vex: separate the pure compiler core (data transformations, no IO) from the binary (CLI argument parsing, filesystem access, temp directory management, Go process invocation, REPL). `lib.rs` already provides `compile()` as a pure function. The split is partially done; the boundary needs to be enforced. + +### Constraint added + +**Resilient parsing.** The parser continues after errors and produces a partial tree. For a Lisp, recovery means skipping to the next top-level form when encountering unbalanced parentheses. + +Why: in an editor, code is almost always in an invalid state mid-edit. A parser that stops at the first error produces no information for the rest of the file — no diagnostics, no type information, no hover. State-of-the-art parsers (Zig, rust-analyzer, tree-sitter) recover and keep going. This is a prerequisite for any future IDE support and costs little to implement for s-expression syntax. + +--- + +## 1. Parametric Polymorphism + +### What Vex has today + +Vex's type system uses concrete types everywhere. `VexType` has `List(Box)`, `Map { key, value }`, `Option(Box)`, and so on — the *containers* are generic in representation, but the *functions that operate on them* are not. + +The type checker handles `map`, `filter`, and `each` as special cases with dedicated methods (`check_map`, `check_filter`, `check_each`). Each method manually extracts the element type from the list, validates the callback signature, and constructs the result type. This pattern repeats for every collection operation that needs to work across element types. + +The `builtins.rs` registry declares fixed signatures: `range` takes `(Int, Int) -> List(Int)`, arithmetic operators take `(Int, Int) -> Int`. Float overloads live in `resolve_call_type` as another special case. + +### Why this matters + +- **Every new collection function requires a new special-case method in the type checker.** Adding `reduce`, `flat_map`, `zip`, `take`, `drop`, or `find` each requires 40-80 lines of hand-written type logic that follows the same structural pattern. +- **User-defined generic functions are impossible.** A Vex user cannot write a function that works on `(List Int)` and `(List String)` with the same definition. The language has generic *types* but not generic *functions*. +- **The type checker carries complexity that belongs in the type system.** The 3,500 lines in `typechecker.rs` are partly a consequence of doing by hand what type variables would do structurally. + +### What state of the art looks like + +**Coalton** (the leading typed Lisp in production) uses Hindley-Milner inference. Coalton 0.2 (March 2026) moved from curried functions to fixed-arity functions specifically because HM inference produced confusing error messages with missing or extra arguments. The field is converging on explicit arities with local inference. + +**Rust** takes a different path — generic type parameters on function signatures are explicit, local variables inside bodies are inferred. No global inference across function boundaries. + +**Go** added generics in 1.18 with explicit type constraints. Go 1.26 (2026) relaxed the recursive type parameter restriction, allowing self-referential constraints like `type Adder[A Adder[A]]`. This strengthens Vex's codegen path — more Vex type patterns can map directly to Go generics. + +### Trade-off: inference vs. explicitness + +Full Hindley-Milner conflicts with Vex's "explicit over clever" philosophy: + +- **Dislocated errors.** When the compiler infers everything, type mismatches surface at the unification point, not at the mistake point. The user sees an error about a type variable they never wrote, at a location far from the actual bug. +- **Opaque code without tooling.** If `(defn foo [x] (+ x 1))` silently infers `(Fn [Int] Int)`, the reader cannot know the type without running the compiler or using an LSP. Vex has no LSP today. +- **Surprising principal types.** HM always finds the most general type. Sometimes that type is more polymorphic than the user intended, and the mismatch surfaces downstream in confusing ways. + +The core need is **parametric polymorphism** (type variables in function signatures), not **global type inference** (omitting annotations). These are separate features that often get conflated. + +### Recommended design + +Require type annotations on `defn` signatures (like Rust). Allow type variables in those signatures. Infer types locally inside function bodies. + +Before (current Vex — `map` as a special-case builtin): + +```vex +;; map only works because the type checker has 80 lines of +;; special-case logic in check_map +(map my-list (fn [x] (+ x 1))) +``` + +After (with parametric polymorphism — `map` as a regular function): + +```vex +(defn map [lst: (List a) f: (Fn [a] b)] -> (List b) + ;; implementation + ) + +(map my-list (fn [x: Int] -> Int (+ x 1))) +``` + +The type checker resolves `a = Int` and `b = Int` from the concrete arguments at the call site — standard **local type argument inference** without requiring the user to write `(map Int Int my-list f)`. + +### What changes in the compiler + +- **`types.rs`**: `VexType::TypeVar(u32)` already exists. Add a `TypeParam { name: String }` variant for named type variables in signatures (`a`, `b`), distinct from anonymous unification variables. +- **`typechecker.rs`**: Add a unification/substitution pass that resolves `TypeParam` to concrete types at call sites. Remove `check_map`, `check_filter`, `check_each` — they become regular generic function calls. +- **`builtins.rs`**: Declare `map`, `filter`, `each` with generic signatures: `(Fn [(List a) (Fn [a] b)] (List b))`. +- **`codegen.rs`**: Go generics (1.18+) map directly — `func Map[A any, B any](lst []A, f func(A) B) []B`. Go 1.26's recursive generics further reduce impedance mismatch for self-referential type patterns. +- **Parser**: No syntax changes needed. `a` and `b` in type position are already parseable as identifiers — the type checker distinguishes them from concrete type names by checking whether they're defined types. + +### What this unlocks + +- User-defined generic functions and data structures +- Collection operations as regular functions, not compiler special cases +- Reduction in type checker complexity (estimated 200-400 lines removed) + +--- + +## 2. Error Propagation (`try` Macro) + +### What Vex has today + +Vex's design principle #3 says "Errors are values — Result types instead of exceptions; errors must be handled or explicitly propagated." The language has `Result` and `Option` types with `match` for destructuring. But there is no propagation mechanism — every `Result`-returning call requires a full `match` to unwrap. + +A typical MCP handler chains multiple fallible operations: + +```vex +(defn handle-search [params: ToolParams] -> (Result JsonValue Error) + (match (validate params) + (Ok validated) (match (db.query validated) + (Ok rows) (match (json.encode rows) + (Ok json) (Ok json) + (Err e) (Err e)) + (Err e) (Err e)) + (Err e) (Err e))) +``` + +Three levels of nesting for three fallible calls. Every `(Err e) (Err e)` arm is pure boilerplate — it re-wraps the error and returns it unchanged. MCP servers are almost entirely I/O, so every handler looks like this. + +### Why this matters + +- **MCP handlers chain 3-5 fallible operations minimum** (parse request, validate input, query/fetch, transform, serialize response). Without propagation, nesting depth grows linearly with the number of fallible calls. +- **The boilerplate obscures the happy path.** The actual logic (`validate → query → encode`) is buried inside match arms. The reader has to mentally filter out identical error-forwarding branches to understand what the function does. +- **Vex's own design principle promises explicit propagation** but does not deliver it. + +### What state of the art looks like + +Every modern language with Result-based error handling provides a propagation mechanism: + +- **Rust** — `?` operator: `let rows = db.query(validated)?;` +- **Zig** — `try` keyword: `const rows = try db.query(validated);` +- **Gleam** — `use` expression: `use rows <- result.try(db.query(validated))` +- **Go** — `if err != nil { return err }` — verbose but explicit + +### Why a simple `try` macro doesn't work + +The obvious design — `(try expr)` expanding to `(match expr (Ok val) val (Err e) (Err e))` — breaks inside `let` bindings. If `expr` returns `Err`, the match evaluates to `(Err e)`, which gets **bound to the variable** instead of returning from the function. Execution continues with an error value where a normal value was expected. + +```vex +;; BROKEN — (Err e) gets bound to `validated`, then db.query receives it +(let [validated (try (validate params)) + rows (try (db.query validated))] + (Ok rows)) +``` + +Without early return semantics, an expression-level `try` cannot propagate errors through `let` bindings. Rust's `?` works because the compiler inserts a `return` — a macro cannot synthesize a `return` that exits the enclosing function. + +### Recommended design: `try` / `catch` + +A `try` macro that takes a binding list and a `catch` clause, expanding into nested `match`: + +```vex +(try [validated (validate params) + rows (db.query validated) + json (json.encode rows)] + (Ok json) + (catch e (Err e))) +``` + +The macro expands this into nested `match`, one level per binding: + +```vex +(match (validate params) + (Err e) (Err e) + (Ok validated) + (match (db.query validated) + (Err e) (Err e) + (Ok rows) + (match (json.encode rows) + (Err e) (Err e) + (Ok json) (Ok json)))) +``` + +If any operation returns `Err`, the `catch` handler executes immediately — no variable binding, no continuation. The macro has access to all bindings, the body, and the catch clause, so it can restructure the entire form. + +### Why `try` / `catch` over alternatives + +Three approaches solve the propagation problem without early return: + +1. **`try-let`** — a combined form that fuses `try` and `let`. Not idiomatic. Lisp favors orthogonal primitives that compose. Fusing two concerns into one compound form means you can't use error propagation outside of `let`, and you can't combine `let` with other effects. + +2. **Gleam-style `use`** — a continuation-capturing form where `use x <- result.try(expr)` rewrites the rest of the block into a callback. More general than `try`/`catch`, but introduces a new syntactic concept (`use`) that Vex users would need to learn. + +3. **Monadic `do` notation** — the fully general solution, works for any monad (`Result`, `Option`, `IO`). Requires higher-kinded types and type classes — heavy prerequisites that Vex doesn't need and isn't planning. + +`try`/`catch` wins because: + +- Every programmer knows the pattern. No new concepts to learn. +- The `catch` clause makes error handling explicit — the reader sees exactly what happens on failure, consistent with "explicit over clever." +- The `catch` clause gives flexibility that Rust's `?` does not: + +```vex +;; Propagate +(catch e (Err e)) + +;; Propagate with context +(catch e (Err (wrap-error e "validation failed"))) + +;; Recover with a default +(catch e default-value) + +;; Log and recover +(catch e (log-error e) empty-list) +``` + +### Single-expression form + +For single fallible operations, `try`/`catch` also works as an expression: + +```vex +(try (parse-int input) + (catch e 0)) +``` + +Expands to: + +```vex +(match (parse-int input) + (Ok val) val + (Err e) 0) +``` + +This form works anywhere — in `let` bindings, function arguments, anywhere an expression fits — because the `catch` clause transforms the `Err` case into a non-`Result` value. The variable receives `0`, not `(Err e)`. + +### What changes in the compiler + +- **`stdlib/prelude.vx`**: Add the `try` macro definition. The macro inspects its arguments: if the first argument is a binding list, expand into nested `match`; if it's a single expression, expand into a flat `match`. The `catch` clause is destructured to extract the error binding name and handler body. +- **No AST, type checker, or codegen changes.** The macro expands to `match` with `Ok`/`Err` patterns — constructs the compiler already handles. + +### Open question: macro complexity + +The binding-list form requires the macro to iterate over pairs and build nested `match` expressions. This is more complex than `cond` (which iterates and nests `if`) but follows the same structural pattern. The single-expression form is trivial — a direct `match` expansion. + +--- + +## 3. Pattern Match Exhaustiveness Checking + +### What Vex has today + +`check_match` in `typechecker.rs` validates that clause bodies have compatible types, but it does not check whether the clauses cover all variants of the scrutinee type. This compiles without any warning or error: + +```vex +(defunion Shape + (Circle Float) + (Square Float) + (Triangle Float Float)) + +(defn describe [s: Shape] -> String + (match s + (Circle r) (str "circle with radius " r) + (Square s) (str "square with side " s))) +``` + +If `s` is a `Triangle` at runtime, the generated Go code hits an unmatched case — either a panic or silent wrong behavior, depending on the codegen. + +### Why this matters + +- **Refactoring becomes unsafe.** Adding a variant to a union should make the compiler flag every match that doesn't handle it. Without exhaustiveness checking, the new variant silently falls through at runtime. +- **It defeats the purpose of static typing.** The type system knows the exact set of variants. Failing to use that information at match sites is leaving value on the table — the compiler has the information to catch the bug and doesn't. +- **`Option` and `Result` are the most common match targets.** Every `(match opt (Some x) ...)` that forgets `None` is a runtime crash that the compiler could prevent. + +### What state of the art looks like + +Every statically typed language with algebraic data types checks exhaustiveness: + +- **Rust** — the `rustc_pattern_analysis` crate implements the full Maranget usefulness algorithm, handling nested patterns, or-patterns, guards, and GADTs +- **Gleam** — uses a decision-tree approach based on Jules Jacobs's pattern matching algorithm, with a dedicated `exhaustiveness.rs` module +- **Elm** — reports missing patterns with concrete examples of unhandled values +- **Zig** — checks exhaustiveness for switch statements on tagged unions and enums + +### Why Vex's case is simpler + +Vex's type system has a property that makes this much easier than in Rust or Gleam: the set of matchable types with known variants is small and fixed. + +Three types have known constructor sets: + +1. **User-defined unions** (`defunion`) — variants declared in the type definition +2. **`Option`** — exactly `Some` and `None` +3. **`Result`** — exactly `Ok` and `Err` + +There are no GADTs, no nested constructor patterns, no or-patterns, no guard-dependent exhaustiveness. Patterns are flat — a constructor pattern binds variables, it doesn't nest constructors inside constructors. + +### Recommended algorithm + +A set-difference check at the end of `check_match`, after all clauses are type-checked: + +1. Collect the set of constructor names covered by the clause patterns +2. If any clause has a `Wildcard` or `Binding` pattern (catches everything), the match is exhaustive — done +3. Get the full variant set from the scrutinee type: + - `VexType::Union { variants, .. }` → all variant names + - `VexType::Option(_)` → `{"Some", "None"}` + - `VexType::Result { .. }` → `{"Ok", "Err"}` + - Primitive types (`Int`, `String`, `Float`) → require a wildcard/binding (can't enumerate all values) + - `Bool` → `{"true", "false"}` (could be special-cased, but requiring a wildcard is sufficient) +4. Compute `missing = full_set - covered_set` +5. If `missing` is non-empty, emit a diagnostic: + - For unions: `"non-exhaustive match: missing variant Triangle of Shape"` + - For `Option`: `"non-exhaustive match: missing None"` + - For `Result`: `"non-exhaustive match: missing Err"` + +### Example diagnostic + +``` +error: non-exhaustive match: missing variant Triangle of Shape + --> src/main.vx:7:3 + | + 7 | (match s + | ^^^^^ + | + = help: add a (Triangle _ _) clause or a wildcard (_) pattern +``` + +### What changes in the compiler + +- **`typechecker.rs`**: Add ~30-50 lines at the end of `check_match`, after the existing clause loop. Walk the checked clauses, collect covered constructor names, compare against the scrutinee type's variant set. +- **No AST, HIR, parser, or codegen changes.** The check runs entirely within the type checker on already-validated pattern information. + +### Limitations of this approach + +This handles Vex's current pattern matching but does not cover: + +- **Nested patterns** — `(Some (Ok x))` matching on `(Option (Result Int String))` would require tracking coverage at each nesting level +- **Or-patterns** — multiple patterns per clause (not in Vex today) +- **Literal coverage** — `(match x 1 "one" 2 "two")` on `Int` requires a wildcard; no attempt to enumerate integers + +If Vex later adds nested constructor patterns, the set-difference approach extends naturally: at each nesting level, collect the covered constructors and check against the type's variant set. The full Maranget usefulness algorithm becomes necessary only with or-patterns or guard-dependent exhaustiveness. + +--- + +## 4. Removal of Traits / Protocols + +### What the design doc had + +§5.4 defined `deftrait` and `impl` syntax. The grammar included `deftrait-form` and `impl-form` as top-level forms. Design principle #2 referenced "algebraic data types and traits." + +### Why traits were removed + +Traits solve the problem of attaching type-specific behavior to types without modifying the type definition — the Expression Problem. This is an object-oriented concern. Functional programming solves the same cases with tools Vex already has: + +- **Higher-order functions** — pass behavior as a function argument instead of requiring the type to implement an interface. `(defn save [item: T to-string: (Fn [T] String)] ...)` lets the caller decide how to serialize without any trait declaration. +- **Union types + pattern matching** — when you control the set of types, model them as variants and match. The exhaustiveness checker (§3) ensures every variant is handled. +- **Parametric polymorphism** (§1) — generic functions that work on any type without constraints. Collection operations like `map` and `filter` don't need trait bounds — the type parameter is unconstrained. + +The concrete use cases from the design doc don't need traits: + +- **Serialization** (`Serializable` trait) — Vex targets Go, where `encoding/json` handles serialization structurally via reflection. A built-in `json.encode` that works on records and unions covers the MCP use case without a trait system. +- **Operator overloading** — Vex has two numeric types (`Int`, `Float`). The compiler handles overloading with a finite dispatch table in `resolve_call_type`. MCP servers don't write generic numeric algorithms. +- **Interface abstraction** — Go uses structural interfaces. Any Go type with the right methods automatically satisfies the interface. Vex can lean on this through `import-go` rather than building a separate nominal trait system. + +The complexity cost of traits is high: + +- Trait resolution algorithm (which impl applies at each call site?) +- Coherence / orphan rules (can module A implement a trait for module B's type?) +- Interaction with type inference (ambiguous type variables when multiple impls exist) +- Go codegen impedance mismatch (Go interfaces are structural, traits are nominal — the mapping is awkward) + +Gleam chose not to have type classes and is a successful production language. Vex follows the same path: if a concrete use case that cannot be solved with higher-order functions or pattern matching arises, traits can be reconsidered from that specific need. + +### What changed + +- **`language-design.md`**: Removed §5.4 (Traits / Protocols), `deftrait-form` and `impl-form` from the grammar, trait references from §2 and §5.1 +- **Compiler**: No changes — `deftrait` and `impl` were never implemented in the parser, AST, type checker, or codegen + +--- + +## 5. Structured Concurrency (`task-group`) + +### What Vex has today + +Vex has two concurrency primitives: `spawn` (fire-and-forget goroutine) and `channel` (Go channel). `spawn` maps directly to `go func() { ... }()` in the generated Go code. There is no mechanism to wait for spawned tasks to complete, propagate errors from child tasks, or scope a set of concurrent tasks to a lifetime. + +The design doc's concurrency section (§10) is four lines of example code and "spawn → goroutine" as the entire model. + +### Why this matters + +The `mcp-go` project (a Go MCP SDK) had a goroutine leak bug in its SSE implementation — when clients disconnected, goroutines waiting on channel sends accumulated until the server exhausted memory. The fix required `context.WithCancel`, `sync.WaitGroup`, and explicit cleanup. This is the class of bug that structured concurrency prevents. + +MCP request handlers commonly fan out concurrent work: + +```vex +(defn handle-query [params: QueryParams] -> (Result Response Error) + (spawn (fetch-user (. params user-id))) + (spawn (fetch-orders (. params user-id))) + ;; how do we wait for both? how do we get their results? + ;; how do we cancel both if one fails? + ) +``` + +With raw `spawn`, there is no answer to any of those questions. The spawned goroutines are detached — the function returns before they complete, and their results are inaccessible. + +### What state of the art looks like + +Every major language has added structured concurrency alongside fire-and-forget: + +- **Kotlin** — `coroutineScope { launch { ... } }` as the default, `GlobalScope.launch` for detached (discouraged) +- **Swift** — `TaskGroup` for scoped work, `Task.detached` for background work +- **Java (JDK 26)** — `StructuredTaskScope` with `fork()` and `join()` +- **Python 3.11** — `asyncio.TaskGroup` in the standard library +- **Go** — `errgroup.Group` as a library (not a language feature) + +None of these removed fire-and-forget. They added structured concurrency as the recommended path for request-scoped work, while keeping detached tasks available for legitimate background work (file watchers, periodic cleanup, long-running listeners). + +### Recommended design + +Keep `spawn` as fire-and-forget. Add `task-group` as a scoped concurrency construct. + +```vex +(defn handle-query [params: QueryParams] -> (Result Response Error) + (task-group [g] + (let [user (spawn g (fetch-user (. params user-id))) + orders (spawn g (fetch-orders (. params user-id)))] + (Ok (build-response (try user) (try orders)))))) +``` + +`task-group` provides: + +- **Scoped lifetime** — all tasks spawned into `g` must complete before the `task-group` body exits +- **Error propagation** — if any task returns `Err`, the group cancels remaining tasks and returns the error +- **Result collection** — spawned tasks return futures that `try` can unwrap + +`spawn` without a group argument remains fire-and-forget for background work: + +```vex +(spawn (watch-file-changes config)) +``` + +### What changes in the compiler + +- **`ast.rs`**: Add `TaskGroup` expression node with a group binding name and body +- **`parser.rs`**: Parse `(task-group [name] body)` as a new special form +- **`typechecker.rs`**: Type-check the group body, track that `spawn` with a group argument returns a future type +- **`codegen.rs`**: Generate `errgroup.Group` with `context.WithCancel`. `spawn g expr` generates `g.Go(func() error { ... })`. The group body ends with an implicit `g.Wait()`. +- **`hir.rs`**: Add corresponding HIR node + +### Trade-off: complexity vs. safety + +This is more compiler work than the `try` macro (which required zero compiler changes). It adds a new AST node, a new HIR node, a new special form in the parser, type checker logic for futures, and Go codegen for `errgroup`. It's a real feature, not syntactic sugar. + +The question is whether the MCP use case justifies it now. MCP servers in practice handle one request at a time with a small number of concurrent operations. The goroutine leak risk is real but manageable with careful use of channels. This feature becomes more valuable as Vex servers scale to handle concurrent sessions with fan-out patterns. + +### Why not a macro + +Unlike `try` (which expands to `match`) and `cond` (which expands to `if`), `task-group` cannot be implemented as a macro. It requires: + +- A new expression type that the type checker understands (futures with typed results) +- Codegen that produces `errgroup.Group` initialization and `g.Wait()` at scope exit +- Context cancellation wiring that has no Vex-level equivalent today + +These are compiler-level concerns, not syntax transformations. + +--- + +## 6. Formatter (`vex fmt`) + +### What Vex has today + +No formatting tool. The user decides indentation, alignment, and line breaks manually. + +### Why this matters + +Every modern language ships a formatter as a first-class tool: + +- **Go** — `gofmt` (2012, day one) +- **Rust** — `rustfmt` (official tool, CI-enforced across the ecosystem) +- **Gleam** — `gleam format` (ships with the compiler) +- **Zig** — `zig fmt` (ships with the compiler) + +A formatter eliminates style discussions, makes code reviews focus on logic, and produces consistent output across projects. For a new language, a formatter signals maturity and reduces friction for new contributors. + +### Why this is simpler for Vex than for most languages + +S-expression formatting has fewer decisions than algol-style formatting: + +- No operator precedence ambiguity — everything is parenthesized +- No semicolons, braces, or optional syntax — indentation follows nesting depth +- No complex expression wrapping rules — a form either fits on one line or each subform gets its own line + +The core algorithm: indent each nesting level by two spaces, keep short forms on one line (under a configurable width), and break long forms with one subform per line. Special-case `defn`, `let`, `match`, `cond`, and `if` with conventional Lisp indentation rules (first argument on the same line as the head). + +### Recommended design + +Ship `vex fmt` as a CLI subcommand. Read `.vx` files, reformat in place (or to stdout with `--check` for CI). Use the existing lexer and parser to produce an AST, then pretty-print from the AST while preserving comments. + +### What changes in the compiler + +- **New file**: `formatter.rs` — AST pretty-printer with comment preservation +- **`main.rs`**: Add `vex fmt` subcommand +- **Lexer/Parser**: No changes, but comments must round-trip (the lexer already tracks comment positions via spans) + +### Constraints + +- Comments must survive formatting — the formatter attaches comments to the nearest AST node and re-emits them +- `vex fmt` must be idempotent — running it twice produces the same output +- The formatter reads from the parser's AST, not from raw text — this guarantees syntactically valid output + +--- + +## 7. Source Location Mapping + +### What Vex has today + +The `--emit-go` flag writes generated Go source for manual inspection. No source location information connects generated Go back to `.vx` source. + +### Why this matters + +When a generated Go program panics, the stack trace shows Go file names, Go line numbers, and Go function names — none of which correspond to what the user wrote. Profiling tools (`pprof`, `go tool trace`) report Go-level locations. Without source mapping, debugging and performance analysis require mentally reverse-engineering the codegen. + +### What state of the art looks like + +- **Go** supports `//line` directives: `//line filename:line` changes the reported source location for subsequent lines in the Go file. The Go compiler, `go vet`, runtime panic traces, and `pprof` all respect these directives. +- **TypeScript** emits `.map` files for JavaScript source mapping +- **Gleam** generates Erlang with source location attributes + +### Recommended design + +Emit `//line` directives in generated Go code. Every HIR node carries a `Span` that maps to the original `.vx` source. The codegen phase resolves each span to a `filename:line` pair and emits a `//line` directive before the corresponding Go code. + +```go +//line main.vx:5 +func HandleSearch(params ToolParams) vexrt.Result[any, error] { +//line main.vx:6 + validated := Validate(params) +``` + +### What changes in the compiler + +- **`codegen.rs`**: Before emitting each Go statement or declaration, emit a `//line` directive using the HIR node's span resolved through the `SourceMap` +- **`lib.rs`**: Pass the `SourceMap` to the codegen phase (currently only the diagnostics formatter uses it) + +### Trade-offs + +- Generated Go becomes harder to read with `//line` directives scattered throughout. The `--emit-go` output for debugging purposes should have an option to suppress directives (`--emit-go --no-line-directives`). +- Go's `//line` directive syntax changed slightly between versions. Target the format supported by Go 1.21+ (Vex's minimum Go version). + +--- + +## 8. Go Toolchain Detection + +### What Vex has today + +`vex build` invokes `go build` and assumes Go is installed and on `PATH`. If Go is missing, the user sees a raw OS error ("command not found" or similar). + +### Why this matters + +Vex's build pipeline requires a Go toolchain. Every Vex user must have Go installed. This is a hard dependency that the installer and CLI should handle gracefully: + +- New users who install Vex via Homebrew or a binary release may not have Go +- The error when Go is missing should explain what is needed and how to get it +- The required Go version (1.21+) should be validated, not assumed + +### What state of the art looks like + +- **Zig** — bundles a C compiler, eliminating the external dependency entirely +- **Gleam** — detects Erlang/Elixir installation and prints clear instructions when missing +- **Dart** — ships a full toolchain in a single SDK download + +Bundling Go is possible (the Go toolchain is a single directory with no global state) but adds ~150MB to the distribution. The lighter approach: detect, validate, and guide. + +### Recommended design + +On `vex build` (before any compilation): + +1. Check if `go` is on `PATH` +2. If missing, print: + ``` + error: Go toolchain not found + + Vex compiles to Go source code and requires the Go toolchain to produce binaries. + + Install Go: https://go.dev/dl/ + macOS: brew install go + Linux: sudo apt install golang (or download from go.dev) + Windows: winget install GoLang.Go + ``` +3. If found, run `go version` and parse the output +4. If the version is below 1.21, print: + ``` + error: Go 1.21 or later required (found go1.18.3) + + Update Go: https://go.dev/dl/ + ``` + +### What changes in the compiler + +- **`main.rs`**: Add a `check_go_toolchain()` function that runs before compilation. Call it at the start of `build` and `run` subcommands. +- No changes to the compiler core — this is a CLI concern. + +--- + +## 9. Summary Extraction Phase + +### What Vex has today + +The compiler pipeline processes one file at a time: lex → parse → expand → type-check → codegen. Multi-file compilation (planned in `docs/dependency-management.md`) will compile dependency modules before the main module, but the pipeline has no explicit step to extract a module's public interface (types and function signatures) separately from type-checking function bodies. + +### Why this matters + +The per-file independence constraint (§0) says each file can be "summarized (exported types and signatures extracted) without reading any other file." But the architecture does not formalize summary extraction as a pipeline phase. + +matklad's "Against Query Based Compilers" (February 2026) describes the map-reduce architecture that Vex's constraints are designed to enable: + +> In parallel, a "summary" is extracted from each file, which is essentially just a list of types and signatures, with function bodies empty. +> +> Sequentially, a "signature evaluation" phase is run on this set of summaries, which turns type references in signatures into actual types, dealing with mutual dependencies between files. This phase is re-run whenever a summary of a file changes. Conversely, changes to the body of any function do not invalidate resolved signatures. +> +> In parallel, every function's body is type-checked. + +This architecture gives two properties: + +- **Parallelism** — function bodies type-check independently once signatures are resolved +- **Incremental invalidation** — changing a function body does not invalidate other files (only signature changes propagate) + +Without a formalized summary phase, multi-file compilation and the future LSP will need to reinvent this boundary ad hoc. + +### Recommended design + +Add a **summary extraction** step between macro expansion and type checking: + +``` +expanded AST → extract_summary() → ModuleSummary +``` + +A `ModuleSummary` contains: + +- Module name +- Exported type definitions (`deftype`, `defunion`) with field/variant types as unresolved `TypeExpr` +- Exported function signatures (name, parameter types, return type) as unresolved `TypeExpr` +- No function bodies + +For single-file compilation, this phase is a no-op pass-through. For multi-file compilation, the pipeline becomes: + +1. Parse + expand all files (parallel, per-file) +2. Extract summaries from all files (parallel, per-file) +3. Resolve signatures across summaries (sequential, cross-file) +4. Type-check function bodies (parallel, per-file, using resolved signatures) +5. Codegen (parallel, per-file) + +### What changes in the compiler + +- **New type**: `ModuleSummary` in `types.rs` or a new `summary.rs` — holds exported names, type definitions, and function signatures +- **`lib.rs`**: Insert `extract_summary()` between `expand()` and `check()` in the pipeline +- **`typechecker.rs`**: Accept a set of `ModuleSummary` values for imported modules when type-checking a file + +### When to implement + +Not needed for single-file compilation. Implement when multi-file compilation begins (`docs/dependency-management.md` §8) — this is the natural point where the summary boundary becomes load-bearing. + +--- + +## 10. LSP Architecture + +### What Vex has today + +No IDE support beyond syntax highlighting (if the user configures a generic Lisp mode). The compiler runs as a batch process. + +### Why this matters + +Gleam v1.14-1.15 (December 2025 — March 2026) shows what drives adoption for a young language: type-directed autocompletion, context-aware compilation, code actions ("add missing type parameter," "merge case branches"), and hover documentation. These features require a language server. + +The per-file independence constraint (§0) and summary extraction phase (§9) exist specifically to enable an LSP without forcing Vex into a query-based architecture. + +### What state of the art looks like + +matklad's "Against Query Based Compilers" (February 2026) recommends pushing queries as late as possible and using direct approaches first. Vex's language properties — explicit imports, no cross-file macros, no glob imports — align with the map-reduce model: + +1. **Per-file work** (parallel): parse, expand macros, extract summary, lower to IR +2. **Cross-file merge** (sequential): resolve signatures from summaries +3. **Per-file work** (parallel): type-check bodies using resolved signatures + +On file change: re-run step 1 for the changed file, diff the new summary against the old one. If the summary changed, re-run steps 2-3. If only the body changed, re-run step 3 for that file only. + +The Ori language (2025-2026) distributes its LSP as a CLI subcommand (`ori lsp`), ensuring version consistency between compiler and language server. Vex should follow this pattern: `vex lsp` launches the language server, using the same compiler binary. + +### Recommended capabilities (ordered by value) + +1. **Diagnostics** — stream type errors and warnings on file save. Requires resilient parsing (§0) and the batch type checker. +2. **Hover** — show resolved types for expressions and function signatures. Requires the HIR with type annotations. +3. **Go to definition** — resolve symbols to their definition site. Requires the type checker's symbol table. +4. **Completions** — suggest names in scope. Requires the type environment at the cursor position. +5. **Formatter** — format on save via `vex fmt` (§6). + +### What changes in the compiler + +- **New file**: `lsp.rs` — LSP server using `tower-lsp` and `lsp-types` crates +- **`main.rs`**: Add `vex lsp` subcommand +- **`lib.rs`**: The `compile()` pipeline already returns diagnostics. The LSP calls the same pipeline functions and streams diagnostics to the editor. +- **Resilient parsing** (§0 constraint): required before the LSP can provide useful results on incomplete code + +### When to implement + +After resilient parsing and summary extraction. The LSP does not need every capability on day one — diagnostics-only is already valuable and validates the architecture. + +--- + +## 11. Tree-sitter Grammar + +### What Vex has today + +No editor syntax highlighting support. Users who want highlighting must configure a generic Lisp mode. + +### Why this matters + +Syntax highlighting is the minimum bar for editor integration. Without it, Vex code looks like plain text. Tree-sitter grammars provide highlighting in Neovim, Helix, Zed, Emacs (tree-sitter mode), and VS Code (via extensions). A tree-sitter grammar gives Vex instant editor support across all major editors with a single implementation. + +### Why this is trivial for Vex + +S-expression syntax maps directly to tree-sitter's grammar DSL. The entire grammar has roughly three rules: + +- **Program** → list of forms +- **Form** → atom | `(` form* `)` | `[` form* `]` | `{` form* `}` +- **Atom** → symbol | keyword | string | number | boolean | nil + +Special forms (`defn`, `let`, `match`, `deftype`, `defunion`, `defmacro`) need field annotations for accurate highlighting (function names, type names, parameter names), adding ~10-15 rules. + +### Recommended design + +Create a `tree-sitter-vex` repository with the grammar definition (`grammar.js`), highlight queries (`queries/highlights.scm`), and installation instructions for each editor. + +### What changes in the compiler + +Nothing — the tree-sitter grammar is an external artifact. It references the same token and syntax rules from `docs/language-design.md` §7 but does not depend on compiler code. + +### When to implement + +Anytime. No dependencies on other roadmap items. High value-to-effort ratio — a few hours of work gives highlighting across all tree-sitter-enabled editors. + +--- + +## 12. Unused Bindings and Import Warnings + +### Why it matters + +Every modern statically typed language warns on unused variables and imports: Rust, Go (errors on unused imports), Gleam, Elm. Without these warnings, dead code accumulates silently: + +- Unused imports inflate the dependency graph and confuse readers +- Unused bindings hide logic errors — a typo in a variable name creates a new binding while the intended one goes unused +- Refactoring becomes uncertain — removing a function or type requires manual search to confirm nothing references it + +For a language that targets MCP server development — where handlers evolve rapidly and prototype code is common — catching dead references early prevents production surprises. + +### State of the art + +- **Go** — unused imports and declared-but-not-used variables are compile **errors**, not warnings +- **Rust** — `#[warn(unused)]` is on by default; unused variables, imports, and functions produce warnings +- **Gleam** — warns on unused variables, unused function arguments, unused imports +- **Elm** — warns on unused imports and unused top-level definitions + +Vex should warn (not error) on unused bindings and imports. Warnings keep the developer informed without blocking compilation during exploratory coding. + +### What to implement + +1. **Unused `let` bindings** — after type-checking a function body, report bindings that were never read. Convention: a leading `_` suppresses the warning (matches Rust/Go) +2. **Unused `import` names** — after macro expansion, report imported names that appear nowhere in the expanded AST +3. **Unused function parameters** — report parameters that are never referenced in the body. Convention: `_` name suppresses the warning + +### Compiler changes required + +- `diagnostics.rs` — add `Severity::Warning` if not already present +- `typechecker.rs` — add a `used: bool` flag to each binding in `Scope`. After checking a body, sweep unused entries and emit warnings. Skip `_`-prefixed names +- A post-macro-expansion pass (or an addition to `typechecker.rs`) — cross-reference imported names against names used in the checked module + +### Trade-offs + +- **Warning vs error**: warnings keep the development loop fast during prototyping. A future `--deny-warnings` flag can promote them to errors in CI +- **False positives in macros**: macro-generated code may reference bindings the user cannot see. Suppressing warnings for compiler-generated names (gensym bindings) avoids noise +- **Performance**: the `used` flag adds one boolean per binding — negligible cost + +--- + +## 13. Installation and Distribution + +### What Vex has today + +No installation mechanism. Users clone the repository and run `cargo build` to produce the compiler binary. There is no release pipeline, no pre-built binaries, and no installer. The CLI has no `--version` flag, no `--help` flag, and no Go toolchain validation — if Go is missing, `vex build` fails with a raw OS error. + +### Why this matters + +A language that cannot be installed in one command cannot attract users. Every modern language ships a frictionless installation path: + +- **Rust** — `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` installs `rustc`, `cargo`, and `rustup` in one step, auto-configures PATH +- **Deno** — `curl -fsSL https://deno.land/install.sh | sh` downloads a single binary +- **Bun** — `curl -fsSL https://bun.sh/install | bash` downloads and auto-configures PATH +- **Go** — official installer at go.dev, plus `brew install go` on macOS +- **Gleam** — `brew install gleam` or pre-built binaries from GitHub Releases + +Vex requires two toolchains (Rust for the compiler, Go for the backend). A user who hears about Vex and wants to try it faces: clone repo → install Rust → `cargo build` → figure out where the binary is → add to PATH → discover they need Go → install Go. Each step is a potential dropout point. + +### Installation design + +The full design lives in `docs/installation.md`. The key components: + +1. **Release pipeline** — a GitHub Actions workflow cross-compiles for 5 targets (macOS x86_64, macOS ARM64, Linux x86_64, Linux ARM64, Windows x86_64) on tag push and publishes a GitHub Release with all artifacts +2. **Installer script** — a POSIX shell script at the repo root that detects OS/arch, downloads the correct binary, installs to `~/.vex/bin/`, and auto-configures PATH +3. **Go toolchain detection** — the CLI validates Go presence and version before compilation with clear error messages +4. **CLI version and help** — `--version`, `-V`, `--help`, `-h`, and `vex version` subcommand + +### Automatic PATH modification + +The installer automatically adds `~/.vex/bin` to the user's shell profile (`.zshrc`, `.bashrc`/`.bash_profile`, or `config.fish`). This is the standard approach for developer toolchains: + +| Installer | Auto-modifies PATH? | How | +|-----------|---------------------|-----| +| **rustup** | Yes (default) | Sources an `env` file from shell rc files | +| **Bun** | Yes (default) | Appends `export PATH` line to rc file | +| **Deno** | No | Prints manual instructions (persistent usability complaint) | + +The pattern is clear: auto-setup with opt-out (`--no-modify-path`) reduces friction for new users without blocking advanced users who manage PATH manually. + +The write is idempotent — the installer checks for `.vex/bin` in the target file before appending. + +### `~/.vex/bin/` as the install directory + +User-local installation (no `sudo`) follows the convention set by rustup (`~/.cargo/bin/`), Deno (`~/.deno/bin/`), and Bun (`~/.bun/bin/`). The `~/.vex/` root co-locates the binary with the global dependency cache (`~/.vex/cache/`, see `dependency-management.md` §6). + +### Go as an explicit dependency + +Three approaches exist for handling the Go toolchain requirement: + +1. **Bundle Go** — ship the Go toolchain inside the Vex distribution (~150MB). Zig does this with its bundled C compiler. Eliminates a separate install step but increases download size dramatically and creates version management complexity (which Go version? how to update it?). + +2. **Detect and guide** — check for Go at build time, print clear error messages with platform-specific install instructions. Gleam follows this pattern for Erlang. Keeps the Vex binary small (~5MB) and lets users manage Go through their preferred method. + +3. **Download on demand** — the first `vex build` automatically downloads Go if missing. Adds significant complexity to the CLI (download progress, version selection, storage location) for a one-time convenience. + +Vex uses option 2: detect and guide. Go has a single canonical installer at go.dev and is available through every major package manager. The barrier is telling the user what they need, not automating the download. + +### Linux static linking + +Linux binaries compile against musl (not glibc) to produce fully static executables. glibc version mismatches cause "GLIBC_2.XX not found" errors on older distributions — the most common class of "the binary doesn't work on my machine" reports for Linux-distributed tools. musl eliminates this entirely at no meaningful performance cost for a compiler binary. + +### Future distribution channels + +These are planned but not yet implemented: + +- **Homebrew** (macOS) — a tap repository with a formula that depends on Go +- **APT / RPM** (Linux) — native packages for Debian/Ubuntu and Fedora/RHEL +- **Scoop / WinGet** (Windows) — package manager manifests +- **Shell completions** — `vex completions bash/zsh/fish` subcommand + +Each channel becomes feasible after the release pipeline produces artifacts. The installer script is the primary channel; package managers are convenience wrappers around the same binaries. + +### Artifact naming convention + +Release artifacts use Go-style `os-arch` naming (`darwin-arm64`, `linux-amd64`) rather than Rust triples (`aarch64-apple-darwin`, `x86_64-unknown-linux-musl`). Vex targets Go developers — the Go naming is familiar to the audience and produces shorter, more readable artifact names on the release page. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..7730d6c --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,243 @@ +# Roadmap + +**Last reviewed:** 2026-03-23 + +This document tracks what Vex needs next, why each item matters, and its current status. Each item links back to `docs/roadmap-rationale.md` for full analysis and trade-off discussion. + +For design constraints that govern all future work, see `docs/roadmap-rationale.md` §0. + +--- + +## Status Legend + + +| Symbol | Meaning | +| ----------- | ------------------------------ | +| Not Started | Work has not begun | +| In Progress | Active development on a branch | +| Done | Merged to main, tested | + + +--- + +## Design Constraint Enforcement + +These are structural changes to align the codebase with the revised design constraints in `docs/roadmap-rationale.md` §0. + + +| Item | Status | Description | +| --------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Enforce compiler core / binary boundary | Not Started | Separate pure compiler core (data transformations, no IO) from the binary (CLI, filesystem, Go process invocation). `lib.rs` already exposes `compile()` — enforce that no IO leaks into the core. | +| Resilient parsing | Not Started | Parser continues after errors and produces a partial tree. For s-expression syntax, recovery means skipping to the next top-level form on unbalanced parentheses. Prerequisite for future IDE support. | +| Summary extraction phase | Not Started | Extract exported types and function signatures from each file independently, before type-checking bodies. Formalizes the per-file independence constraint as a pipeline step. Prerequisite for multi-file compilation and LSP. See `roadmap-rationale.md` §9. | + + +--- + +## Type System + + +| Item | Status | Rationale reference | +| ----------------------- | ----------- | ------------------------- | +| Parametric polymorphism | Not Started | `roadmap-rationale.md` §1 | + + +### Parametric Polymorphism — Summary + +The type system has generic containers (`List(Box)`, `Map { key, value }`, `Option`) but no generic functions. Collection operations (`map`, `filter`, `each`) are special-cased in the type checker. Adding type variables to function signatures enables: + +- User-defined generic functions +- Collection operations as regular functions (removes ~200-400 lines of special-case type checker code) +- Foundation for future type classes / traits + +**Compiler changes required:** + +- `types.rs` — add `TypeParam { name: String }` variant for named type variables in signatures +- `typechecker.rs` — add unification/substitution pass; remove `check_map`, `check_filter`, `check_each` +- `builtins.rs` — declare `map`, `filter`, `each` with generic signatures +- `codegen.rs` — generate Go generics (1.18+; Go 1.26 recursive generics reduce impedance mismatch) +- Parser — no changes needed (lowercase identifiers in type position already parseable) + +--- + +## Error Handling + + +| Item | Status | Rationale reference | +| ------------------------------------- | ----------- | ------------------------- | +| Error propagation (`try` / `catch`) | Not Started | `roadmap-rationale.md` §2 | +| Pattern match exhaustiveness checking | Not Started | `roadmap-rationale.md` §3 | + + +### Error Propagation — Summary + +Vex has `Result` and `Option` types but no propagation mechanism. Every fallible call requires a full `match` with boilerplate `(Err e) (Err e)` arms. MCP handlers chain 3-5 fallible operations, creating deep nesting. + +A `try` / `catch` macro solves this with two forms: + +- **Block form** — takes a binding list and a `catch` clause, expands into nested `match`: +`(try [x (op1) y (op2 x)] (Ok y) (catch e (Err e)))` +- **Expression form** — single operation with recovery: +`(try (parse-int s) (catch e 0))` + +**Compiler changes required:** + +- `stdlib/prelude.vx` — add the `try` macro definition +- No AST, type checker, or codegen changes needed + +### Exhaustiveness Checking — Summary + +`match` does not verify that all variants of a union, `Option`, or `Result` are covered. Missing variants cause runtime failures that the compiler has enough information to catch at compile time. + +**Compiler changes required:** + +- `typechecker.rs` — add ~30-50 lines at the end of `check_match` to compare covered constructors against the scrutinee type's variant set + +--- + +## Diagnostics + + +| Item | Status | Rationale reference | +| --------------------------------------- | ----------- | -------------------------- | +| Unused bindings / imports warnings | Not Started | `roadmap-rationale.md` §12 | + + +### Unused Bindings / Imports — Summary + +The compiler accepts `let` bindings and `import` declarations that are never referenced. This makes dead code invisible and slows refactoring — the developer has no signal that an import or binding can be removed. + +**Compiler changes required:** + +- `typechecker.rs` — track a `used: bool` flag per binding in `Scope`. After checking a function body, emit a warning diagnostic for every unused entry +- `macro_expand.rs` or a post-expansion pass — track which imported names appear in the expanded AST. Emit warnings for unreferenced imports +- `diagnostics.rs` — add `Severity::Warning` (currently only `Error` exists) + +--- + +## Concurrency + + +| Item | Status | Rationale reference | +| ------------------------------- | ----------- | ------------------------- | +| Structured concurrency (`task-group`) | Not Started | `roadmap-rationale.md` §5 | + + +### Structured Concurrency — Summary + +Vex has `spawn` (fire-and-forget goroutine) and `channel` (Go channel) but no mechanism to scope concurrent tasks, wait for completion, or propagate errors from child tasks. MCP handlers that fan out concurrent work have no way to collect results or cancel on failure. + +`task-group` adds scoped concurrency: all tasks spawned into a group must complete before the group exits, errors cancel remaining tasks, and spawned tasks return futures. + +**Compiler changes required:** + +- `ast.rs` / `hir.rs` — add `TaskGroup` expression node +- `parser.rs` — parse `(task-group [name] body)` as a special form +- `typechecker.rs` — type-check group body, track that `spawn` with a group argument returns a future type +- `codegen.rs` — generate `errgroup.Group` with `context.WithCancel` + +--- + +## Developer Experience + +Items from `docs/compiler-architecture.md` §13 and state-of-the-art gaps identified in `docs/roadmap-rationale.md`. + + +| Item | Priority | Status | Rationale reference | Description | +| ----------------------------- | -------- | ----------- | -------------------------------- | ------------------------------------------------------------------------------ | +| `vex fmt` (formatter) | P1 | Not Started | `roadmap-rationale.md` §6 | Opinionated code formatter for `.vx` files, shipped as a CLI subcommand | +| `vex dev` (hot reload) | P1 | Not Started | `compiler-architecture.md` §13 | File watcher that recompiles and restarts on source changes | +| Structured logging | P1 | Not Started | `compiler-architecture.md` §13 | Key-value structured log output via `vex.log` stdlib | +| Source location mapping | P1 | Not Started | `roadmap-rationale.md` §7 | Emit `//line` directives in generated Go so stack traces point to `.vx` source | +| Go toolchain detection | P1 | Not Started | `roadmap-rationale.md` §8 | Validate Go installation and version before compilation with clear error messages | +| Connected REPL | P2 | Not Started | `compiler-architecture.md` §13 | REPL that connects to a running `vex dev` process (nREPL model) | +| Error chain traces | P2 | Not Started | `compiler-architecture.md` §13 | Full chain display when `Result` errors propagate through multiple functions | +| Test framework | P3 | Not Started | `compiler-architecture.md` §13 | Assertions, test discovery, test runner via `vex.test` stdlib | + + +--- + +## IDE Support + +Editor tooling, from lightweight (tree-sitter) to full (LSP). Each item builds on the one above. + + +| Item | Priority | Status | Rationale reference | Description | +| --------------------- | -------- | ----------- | -------------------------- | -------------------------------------------------------------------------------------------- | +| Tree-sitter grammar | P1 | Not Started | `roadmap-rationale.md` §11 | `tree-sitter-vex` grammar for syntax highlighting in Neovim, Helix, Zed, Emacs, and VS Code | +| `vex lsp` (LSP) | P2 | Not Started | `roadmap-rationale.md` §10 | Language server shipped as a CLI subcommand, starting with diagnostics and hover | + + +### Tree-sitter Grammar — Summary + +S-expression syntax maps directly to tree-sitter's grammar DSL (~15-20 rules). Provides instant syntax highlighting across all tree-sitter-enabled editors. No compiler dependencies — can be built anytime. + +### LSP — Summary + +The per-file independence constraint (§0) and summary extraction phase enable a map-reduce LSP architecture without query-based complexity (Salsa, rust-analyzer). The architecture follows matklad's recommendation: parse/expand/summarize per-file in parallel, merge summaries sequentially, type-check bodies per-file in parallel. On file change, re-run only the changed file's pipeline; propagate only if the summary changed. + +**Prerequisites:** resilient parsing, summary extraction phase + +**Capabilities (ordered by implementation priority):** + +1. Diagnostics — stream errors/warnings on file save +2. Hover — show resolved types for expressions +3. Go to definition — resolve symbols to definition sites +4. Completions — suggest names in scope +5. Format on save — via `vex fmt` + +--- + +## Installation and Distribution + +Getting Vex onto a user's machine and making it work from the terminal. Full design in `docs/installation.md`. Rationale in `roadmap-rationale.md` §13. + + +| Item | Priority | Status | Description | +| ----------------------------- | -------- | ----------- | ------------------------------------------------------------------------------------------------ | +| CLI version and help flags | P0 | Not Started | `--version`, `-V`, `--help`, `-h`, and `vex version` subcommand | +| Go toolchain detection | P0 | Not Started | Validate Go installation and version (>= 1.21) before compilation with clear error messages | +| Release pipeline | P0 | Not Started | GitHub Actions workflow that cross-compiles 5 targets on tag push and creates a GitHub Release | +| Installer script | P0 | Not Started | `curl \| sh` installer that downloads the binary, installs to `~/.vex/bin/`, auto-configures PATH | +| Homebrew formula | P2 | Not Started | Homebrew tap for `brew install vex` on macOS | +| APT / RPM packages | P3 | Not Started | Native Linux packages with Go as a declared dependency | +| Scoop / WinGet manifests | P3 | Not Started | Windows package manager support | +| Shell completions | P3 | Not Started | `vex completions bash/zsh/fish` subcommand for tab completion | + + +--- + +## MCP Framework + +The end goal of Vex. These items build on top of the language-level features above. + + +| Item | Status | Description | +| ---------------------------------------------- | ----------- | -------------------------------------------------------------------------------- | +| `deftool` / `defresource` / `serve-mcp` macros | Not Started | Core MCP server authoring macros | +| Request/response tracing | Not Started | Logging all incoming JSON-RPC requests and outgoing responses | +| MCP-aware test utilities | Not Started | Mock MCP client, JSON schema validation, session lifecycle simulation | +| Transport dev mode | Not Started | Auto-configuration of stdio and Streamable HTTP transports | +| Session management | Not Started | Session ID handling, Origin header validation per the 2025-11-25 MCP spec update | +| Protocol error diagnostics | Not Started | Mapping Vex `Result` errors to JSON-RPC error codes | + + +--- + +## Completed Milestones + + +| Milestone | Date | +| ------------------------------------------------------------- | ---- | +| MVP — hello world, fibonacci, fizzbuzz compile and run | Done | +| Records (`deftype`) and field access | Done | +| Unions (`defunion`) and pattern matching (`match`) | Done | +| `Result` / `Option` types | Done | +| Collections — `List`, `Map`, `each`, `range`, `map`, `filter` | Done | +| Modules — `module`, `export`, `import` | Done | +| Go interop — `import-go` | Done | +| Concurrency — `spawn`, `channel`, `send`, `recv` | Done | +| REPL — tree-walking interpreter | Done | +| Self-hosted macros — `defmacro` with automatic hygiene | Done | + +