performance benchmark and optimization

[t41372]

skit should be able to call itself lightweight and point at numbers. Right now it can't: there's no
reproducible measurement, so any performance claim would be prose.

Two halves: build the measurement, then earn the numbers. PR #29 is the first half and changes
nothing under src/skit/. This issue tracks the whole thing.

Baseline

Nightly artifact on 64156e1 (ubuntu-24.04, python 3.13.14, profile full):

skit --version	154 ms (+139 ms over python -c pass)
list --json @0 / @100 / @1000	142 / 199 / 653 ms → 512 µs/entry
modules imported by --version	291, including typer and rich
modules imported by list @0 / @100	291 / 315 (the second loads three tree-sitter grammars)
peak RSS --version / list@1000 / TUI@1000	30 / 68 / 49 MiB
TUI first idle @1000	395 ms
file syscalls, list --json @1000	12029
wheel / installed closure	508 KB / 19.2 MB

Hosted-runner wall clock swings ~25% depending on which CPU the runner draws, so times are target
tier, not gates. The module census and wheel size are deterministic and already enforced.

Violated target budgets today — this is the backlog:
startup.version.over_python_ms, scale.list_json.per_entry_us, rss.list_json.n1000.peak_kib,
imports.version.has_typer, imports.version.has_rich, imports.list_json.n100.has_tree_sitter.

Findings

1. Listing a library imports native parsers. spec_for() builds a full LangSpec on first
resolve, which imports that language's grammar:

$ python -c "from skit.langs.registry import spec_for; spec_for('js')"
# imports tree_sitter_javascript, tree_sitter_typescript

An empty library lists at 291 modules with no grammar loaded; 100 entries lists at 315 with
tree_sitter, tree_sitter_bash and tree_sitter_javascript resident. The tree-sitter stack alone
imports in ~56 ms (local, indicative). The TUI list resolves kinds per row too, for the glyph and
mode badge.

2. The analyzer runs when a launch menu opens. flows calls analyzer.reconcile() while
building the form, so opening an entry re-parses its script: 55 ms for a 2000-line python or shell
script, ~57 ms more if it's the first parse in the process (micro.analyze*). The TUI suite
measures first-idle and search, so this path is unmeasured.

3. Per-script JS dependencies are unmeasured. launch._ensure_deps() materializes a
node_modules next to the stored copy on run. footprint covers wheel, sdist, closure, installed
package and distribution count — nothing about what a user's library weighs after a few JS entries.

4. i18n is not a startup cost. Non-English locales parse a ~100 KB .mo on first gettext():
~0.5 ms. English ships no catalog. Nothing to do here.

Plan

Measurement pipeline — PR #29

• benchmarks/ package: suites, deterministic dataset generator, results schema, two-tier
  budget contract with a ratchet protocol, A/B compare tooling
• PR / nightly / on-demand-compare workflows, pinned hyperfine, 90-day result artifacts
• pipeline logic under the repo's 100% coverage floor; only spawn-and-wait orchestration exempt
• baseline captured from CI and ratchet bounds proposed from that artifact
• import census at a populated N as well as an empty one (finding 1 — an N=0-only census
  reported has_tree_sitter = 0 on the one input where it could not fail)
• post-merge: dispatch the nightly once, to confirm the profiles = ["full"] enforced rows
  evaluate green (schedule-only workflows never run pre-merge) — run on 64156e1, 5/5
  enforced rows passed, no skips; it is the baseline above

P0 — close the remaining measurement gaps

Pipeline only, no behavior change. All additive metric IDs, so they can land after #29 without
breaking budget keys or history series.

• TUI spans for selection change and launch-menu open (finding 2)
• footprint: user-library weight and per-script node_modules, reported apart from core
  (finding 3)
• analyzer micro: a syntax-error / half-written source variant — the common case for a launcher

P1 — CLI fast path

• tiny skit.__main__:main dispatcher; --version without typer or rich
• move analysis / healthcheck / flows into the command bodies that use them
• accept: imports.version.has_typer and has_rich reach 0 and are promoted to enforced in
  the same PR; census ratchet re-proposed from that PR's artifact;
  startup.version.over_python_ms at least halved, and the --version instruction count
  drops by a comparable margin (the deterministic half of that claim)

P2 — kind display without parsers

• parser-free KindDescriptor (glyph, mode support, family) for list and TUI display;
  spec_for() stays for analysis and launch
• accept: imports.list_json.n100.has_tree_sitter reaches 0, promoted to enforced

P3 — library summary index

• EntrySummary in registry.toml plus one activity index; full meta.toml read only on
  select / run / edit, registry still rebuildable, corruption still falls back
• accept: scale.list_json.per_entry_us at least halved (250 µs target);
  syscalls.list_json.file_ops drops proportionally at N=1000

P4 — analyzer and memory

• reuse parser objects across a form's reconcile
• Memray flamegraphs for import / list / TUI
• decide from the closure numbers whether tree-sitter should become an extra
• accept: the launch-menu-open span improves at 2000 lines; rss.list_json.n1000.peak_kib
  under its 64 MiB target

P5 — ratify the contract, then publish the numbers

The user-facing numbers are the only performance history anyone outside the repo reads, and
they are worth publishing once, after the optimization work has settled — not kept live.
Nothing here is urgent, and doing it early means publishing numbers we intend to beat.

• promote surviving target rows to enforced where they're ratchet-safe
• README performance section generated from a results.json (commit, date, host, run counts),
  never hand-written, refreshed on releases rather than continuously
• wording: "skit meets its published budgets on the reference system", not "skit is lightweight
  because the wheel is small"

CodSpeed — PR #32

Worth adding for one reason: simulation mode measures CPU instructions rather than wall clock, so
it's hardware-independent with <1% variance. Every timing budget here is stuck at target tier
because hosted-runner medians move with whatever CPU the runner drew — the two runs behind the
baseline above differed by 26% on identical code — which means nothing that matters can fail a PR
today. Instruction counts can. The CodSpeed CLI also runs arbitrary executables
(codspeed exec --mode simulation -- skit --version), so CLI startup is covered, not just
in-process Python.

Free for public repos, and public repos need no CODSPEED_TOKEN secret at all (OIDC since
2025-11), so fork PRs work too and nothing here needs pull_request_target.

After #29 rather than inside it: CodSpeed reports a base-vs-head delta, so it has nothing to say
until there's a run on main to compare against. Its first useful report is the first PR after the
pipeline merges.

• setup: CodSpeed app installed, OIDC, no repo secret
• pytest-codspeed over in-process hot paths: per-language analyze() at 20/200/2000 lines,
  store.list_entries / resolve over a 200-entry library
• keep hyperfine / pyperf: self-contained, no account needed, and the source of the metrics
  CodSpeed's instruments cannot take (wheel bytes, closure bytes, module censuses, syscall
  counts). CodSpeed owns timing trends; budgets.toml stays the contract
• amend the "no SaaS onboarding" non-goal in docs/design/benchmarks.md, which this reverses
• codspeed exec --mode simulation -- skit --version / skit list --json: CLI startup as an
  instruction count. ci: add CodSpeed continuous performance measurement #32 covers in-process callables only, so the one path P1 is about is still
  measured by wall clock alone — see below

Related

#16 (single binary) changes the footprint story, but most of the startup cost measured here is
import graph rather than interpreter boot, so it should be decided after P1–P3.