Skip to content

Offline typeshed: store-backed pins, a segregated download crate, and checking that never fetches#338

Merged
MelbourneDeveloper merged 106 commits into
mainfrom
typeshed
Jul 21, 2026
Merged

Offline typeshed: store-backed pins, a segregated download crate, and checking that never fetches#338
MelbourneDeveloper merged 106 commits into
mainfrom
typeshed

Conversation

@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

TLDR

Typeshed acquisition splits into an offline, store-backed path used exclusively by the checker/LSP/MCP and a segregated basilisk-typeshed-fetch crate that runs only on explicit user action (basilisk typeshed download, or the editor's Download buttons), so checking never touches the network.

What Was Added?

  • Content-addressed storebasilisk-stubs/src/typeshed/store.rs ([STUBRES-TYPESHED-STORE] / [STUBRES-TYPESHED-PIN]): one immutable directory per commit; read_snapshot hashes commit-object, obtains its root-tree OID via gittree::commit_root_tree (which is where the tree <40-hex> header parsing lives), and re-hashes the materialized bytes plus manifest rows into Git trees, requiring the root to match.
  • The only network crate — new crates/basilisk-typeshed-fetch (sole owner of ureq; [STUBRES-TYPESHED-DOWNLOAD] / [TYPESHEDRT-SEGREGATION]): github.rs confines the client and its bearer token to api.github.com/codeload.github.com; commit.rs rebuilds GitHub's signed commit payload into an object that must hash to the requested SHA.
  • User-invoked surfacesbasilisk-cli/src/typeshed_cli.rs adds basilisk typeshed download [--commit <sha>] (exit 0/2/3), writing typeshed-commit and retiring typeshed-path in one apply_config_patch transaction; basilisk-lsp/src/typeshed_download.rs is the only LSP code that fetches typeshed, and it delegates the transfer itself to basilisk-typeshed-fetch (the LSP crate declares no HTTP client).
  • Offline resolution and hoverconfiguration_editor/typeshed_resolution.rs (replacing the deleted 406-line typeshed_acquisition.rs) builds a terminal TypeshedGeneration from a local read; new hover/access.rs + hover/render.rs add receiver classification and one SymbolCard renderer ([LSPARCH-FEATURES-HOVER]).
  • Build gatesscripts/check-dependency-shape.sh fails if basilisk-stubs/basilisk-checker resolve any HTTP client ([STUBRES-TYPESHED-OFFLINE]); scripts/gen_readmes.py ([README]) renders all five READMEs behind a new _lint_docs target; gen_rules_reference.py adds per-rule references ([WEBSITE-ERROR-PAGES-REFERENCES]).

What Was Changed or Deleted?

  • Two sources onlySourceSelection::{Custom, Pinned{commit, explicit}} replaces {Custom, ExactCommit, Latest}; SourceKind::Latest, the basilisk-stubs Transport and Provenance types and TypeshedStatus.{transport, provenance, signed_release} are gone (unrelated same-named items — the basilisk-cli LSP-transport enum, TagKind::Provenance, TypeProvenance — are untouched); GateConfig.verify_content is removed so the Content gate always runs.
  • Network and cache deletedtypeshed/cache.rs (682 lines: DiskCache, the 24-hour TTL), typeshed/transport.rs and its http submodule (mirrors are gone entirely); runtime.rs shrinks from ~490 lines to RuntimeBackend { store_root } with an infallible production_manager.
  • Three config keystypeshed-url, typeshed-cache, typeshed-cache-path and typeshed-verify leave BasiliskConfig, the parser and the TypeshedConfigKey allowlist, replaced by typeshed-store-path; TypeshedConfigValue and is_valid_typeshed_url_template are deleted.
  • Checking never downloads — a pin absent from the store is a terminal NO SOURCE PipelineError::Internal and main.rs exits 3 (was 1); TypeshedGeneration loses Acquiring/Blocked for Ready/NoSource, init.rs resolves typeshed to a terminal generation during initialize rather than transitioning through an acquiring state, and apply_prepared_patch/watch.rs drop rollback branching.
  • Hover, panel and mechanical churnpush_reference_sections resolves members only through their receiver; the #state-overlay lock screen becomes an inline #state-notice row, and the typeshed editor gains noSourceRow() plus per-button markBusy() spinners. 201 rule files change one header line each (spec anchors uppercased), no logic change.
  • Untested Neovim suites are now gatedscripts/test-nvim.sh gains run_plenary_dir and runs tests/basilisk (12 specs) and tests/dap before the LSP e2e pass. Those 15 spec files were previously executed by nothing at all: not the Makefile, not any script, not any workflow. They are gated identically, through assert_plenary_pass ([LSPTEST-EDITOR-SPECIFIC-INTEGRATION-NEOVIM-E2E-GATE]).
  • basilisk-cli reported the wrong exit code for an internal failure — a panic in the analysis thread returned 1 ("error diagnostics found"), telling a CI consumer that the user's code had findings when none had been produced. It now returns 3 per [CHKARCH-CLI-EXITCODES].
  • configuration-editor.ts was over the 500-LOC ceiling (534 lines on main, 527 at this branch's head), breaking both that ceiling and [VSIX-CONFIGURATION-EDITOR-FILES]'s explicit claim that every file in that group is under it. The LSP seam — capability probe, ConfigurationEditorTransport, the six request-method constants, workspace-root selection — moves to configuration-editor-transport.ts, leaving 452 + 104 lines, re-exported so no importer changed.

How Do The Automated Tests Prove It Works?

  • Every tamper breaks the chaintypeshed/store.rs: write_then_read_verifies_the_full_offline_chain, any_mutated_byte_fails_the_pin, a_tampered_commit_object_fails_before_its_tree_is_trusted, interrupted_staging_never_becomes_an_entry; gittree.rs::commit_oid_matches_git pins the constant against git hash-object -t commit.
  • Download writes nothing unless the proof holdsbasilisk-typeshed-fetch/src/tests.rs::{every_transport_failure_writes_nothing, tampered_archive_bytes_fail_content_binding_and_write_nothing}; commit.rs::a_real_signed_typeshed_commit_reconstructs_to_its_published_sha uses a real signed API response; github.rs proves the credential never leaves official hosts nor appears in Debug.
  • A missing pin fails hard instead of fetchingpipeline/tests.rs::a_missing_pin_tanks_the_check_instead_of_downloading asserts NO SOURCE, the recovery command and zero store writes; the rewritten runtime/tests.rs adds a_pin_absent_from_this_machine_is_terminal_no_source and a_corrupt_store_entry_is_terminal_no_source.
  • Retired surfaces stay retiredtypeshed_forbidden_policy_tests.rs asserts ureq/reqwest/hyper never appear in the basilisk-stubs manifest; content_verification_cannot_be_waived replaces waived_content_emits_unverified_only; check_cli_rejects_retired_typeshed_waiver_flags proves the dropped flags fail to parse; generation_states_are_terminal_ready_or_no_source serde-proves Acquiring/Blocked are undeserializable.
  • The write path, hover and the paneltypeshed_cli.rs::a_failed_latest_download_writes_no_pin; hover_member_resolution_tests.rs::hover_on_member_never_answers_from_an_unrelated_bare_module_name; configuration-editor-typeshed-dom.test.ts adds assertNothingLocked (no overlay, no disabled control) and assertEveryPostedIntentDecodes(), which caught the webview posting a { kind: 'Text', value } wrapper that the decoder silently drops — the webview now sends a bare string.

Full CI was run locally on this branch before submission: fast gates, website (build + Playwright e2e), zed, test-rust (per-crate coverage + the live python/typing conformance gate), lint (clippy + eslint + deslop + cargo-audit), verify-binaries (shipwright manifest/versions + release-attribution policy), build, test-nvim (unit + DAP + LSP e2e + screenshots + luacov threshold), test-vscode (617 passing, coverage 93% ≥ 92%), and the mutation gate — all green.

Spec / Doc Changes

  • CHECKER-STUB-RESOLUTION-SPEC.md — new [STUBRES-TYPESHED-OFFLINE], [STUBRES-TYPESHED-PIN], [STUBRES-TYPESHED-STORE], [STUBRES-TYPESHED-DOWNLOAD]; [STUBRES-TYPESHED-ACQUIRE] is deleted and retargeted in CHECKER-CACHE-SPEC; [STUBRES-TYPESHED-WARN] rewritten to UNPINNED / USER-MANAGED SOURCE / LICENSE CHANGED / NO SOURCE (exit 3).
  • Plan and protocol specsCHECKER-TYPESHED-RUNTIME-PLAN.md gains the [TYPESHEDRT-SEGREGATION] crate table asserted by check-dependency-shape.sh; LSP-CONFIGURATION-EDITOR-SPEC.md gains [LSPCFGED-TYPESHED-DOWNLOAD]; [LSPARCH-FEATURES-HOVER] gains a normative free-name vs member-access rule. New: DOCS-README-SPEC.md ([README]), REPO-STANDARDS-SPEC.md ([REPO-STANDARDS]).
  • Corrections and generated docs[ANALYSIS-CAPS] now advertises only workspace/willRenameFiles, and the VSIX/ZED/NEOVIM/LSP-PROFILING specs plus crate READMEs are corrected to shipping layouts. All five READMEs are generated from docs/readme/README(.zh).src.md and gain a "Standard-library types, always offline" section.
  • Three false claims removed. (1) CLAUDE.md and [CHKARCH-PERF-PARALLEL] described "file-level parallelism using Rayon (work-stealing)" — no Basilisk crate calls Rayon; there are zero par_iter/rayon:: uses across crates/, and rayon reaches the lockfile only transitively via salsa and ruff_db. Both now describe what runs: single-threaded analysis on one large-stack thread ([LSPARCH-ARCH-STACK]), with Tokio concurrency confined to the LSP server. (2) [VSIX-CONFIGURATION-EDITOR-FILES] claimed every file was under 500 LOC; one was not, and is now split. (3) VSIX-SPEC claimed "every memory command is gated on basilisk.debugging" — 7 of 9 are: memoryTrackProcess is "when": false, and trackMemoryCurrentFile is deliberately ungated because it starts the session.

Breaking Changes

  • Yes — see below.

  • Checking never downloads. basilisk check/analyze, the LSP and MCP no longer fetch typeshed. A pin absent from the local store fails with exit 3 and NO SOURCE — <sha> is not on this machine; run Download latest or basilisk typeshed download --commit <sha>, with no bundle or main fallback. An unset typeshed-commit now resolves to the bundled commit as an implicit pin instead of tracking latest.

  • Config and CLI. typeshed-url, typeshed-cache, typeshed-cache-path and typeshed-verify are removed and silently ignored; typeshed-store-path supersedes typeshed-cache-path and typeshed-store-path = false is a hard config error. --no-typeshed-cache and --no-typeshed-verification now cause a parse error.

  • Build requirements. _lint_rust now runs scripts/check-dependency-shape.sh and fails if basilisk-stubs/basilisk-checker resolve an HTTP client, and make lint gains a _lint_docs generated-docs drift gate. Hand-editing any generated README fails make lint and the CI website job — edit docs/readme/*.src.md and rerun gen_readmes.py.

  • Wire protocols — editor clients must update in lockstep. TypeshedStatusState drops transport, provenance, signedRelease and blockedReason for noSourceReason; Acquiring/BlockedDownloading/NoSource; PinCurrent/AcquireFreshDownloadLatest/DownloadPinned; the TypeshedSettingValue enum (Text/Boolean) is removed and SetTypeshedSetting now carries a bare String, so a client still posting the tagged {kind:'Text'|'Boolean'} shape has its edit silently dropped by the decoder. MCP declares openWorldHint: false.

  • Public Rust API. Removed: AcquisitionBackend, Transport/HttpsTransport, DiskCache, Provenance, TypeshedConfigValue, is_valid_typeshed_url_template, TypeshedWarning::{DownloadFailed, Unverified}, GateConfig.verify_content. Changed: production_manager takes one TypeshedRequest and is infallible; ExternalSymbol/ExternalMethod gained docstring, so struct-literal sites must be updated.

- The LSP watches every root's config server-side (250ms content poll of
  pyproject.toml/uv.lock/.python-version) and re-checks, republishes
  diagnostics, and pushes basilisk/configurationChanged on any change,
  whether from a file edit or the configuration-editor UI
- The in-memory root config is authoritative over disk: per-file discovery
  walks only strictly below the owning root (load_basilisk_config_below),
  so an applied UI change is never clobbered by stale on-disk content
- basilisk format [paths] [--check] via the embedded Ruff formatter, plus a
  generated release-notes component block from shipwright.json and the
  release binary itself
- Guards: disk-edit and UI-apply reactivity e2e tests, stale-disk authority
  unit test, empty-PATH format CLI e2e, release-notes drift test, and nvim
  client-module specs (Lua coverage 45% >= 44%)
- Coverage ratchets: basilisk-lsp 83->84, basilisk-config 96->97
…le types (Refs #317)

- BSK-0001 skips parameters whose literal default determines the type
  ([TYPEINF-FUNC-DEFAULTS]); BSK-0002 skips functions whose return type the
  engine already infers ([TYPEINF-FUNC-RETURN]); exemptions exactly as strong
  as the current inference engine (shared rhs_fully_determines_type)
- Specs updated: CHECKER-ARCHITECTURE-SPEC, CHECKER-TYPE-INFERENCE-SPEC
- CLI adopt/fix and LSP autofix/mass-fix follow the same predicate
- Tests updated across checker, CLI e2e fixtures, and LSP e2e to enforce
  the new behavior in both directions
- Mutation baseline refreshed by the full 142-mutant run (0 missed, 100% kill)
- Benchmark CSV re-established under the post-#313 config-neutral harness
  (see PR body: no code regression; the old baseline predates the harness change)
…g testing

1aa6e0e replaced the entire wheel packaging definition (maturin build-system,
basilisk-python project metadata, bundled license notices) with a scratch
[tool.basilisk.rule-tags] config that also downgraded pep rules to warnings.
The wheel is how the python/typing conformance suite installs checkers via
uv sync, and the release pipeline builds it — restore main's file verbatim.
Verified: config_override_tests, rule_tags_tests, basilisk-cli bin tests all pass.
# Conflicts:
#	crates/basilisk-lsp/src/server/activity_panel/module_tree.rs
#	mutation_testing/mutants_report.html
#	mutation_testing/mutation_scores.json
#	vscode-extension/src/module-explorer.ts
…tion-level Salsa queries

Stage 0 (NARROWPLAN-CHECKLIST): two-mode bidirectional core (synth/check,
check primary) over the ruff AST; two-stage constraint architecture with a
separate solver delegating every ground leaf to is_assignable_to; bounded
polar type variables with deferred generalization (list[Var{lower=Literal[1]}]);
gradual-guarantee differential harness (curated corpus + conformance-fixture
sweep); prototype validation against five pinned real typeshed stubs.

Stage 1: definition/expression-level tracked queries with slice-keyed early
cutoff (proven via WillExecute log), compact module-interface boundary with
backdating, salsa fixpoint cycles seeded by the Unknown sentinel with a hard
cap. Measured on a seeded 1.12M-LOC corpus: keystroke re-check p50 0.16ms /
p99 0.22ms, RSS ~312MB.

Fixes found by the new gradual harness and reflexivity validation, all
conformance-verified (100%, 0 FP, CSV byte-identical): classes_override no
longer treats an absent annotation as a signature mismatch; is_assignable_to
gains the documented-but-missing bool <: int <: float arm; union-left
decomposition now precedes Optional-target unwrapping so A | None <: Optional[B]
holds. types.rs split under the 500-line cap (types_star_tuples.rs).

Refs #317
…erations

First consumer of the resolver's collected NarrowingGuards
(TYPEINF-TARGET-NARROWING): intersection-and-negation set operations over
InferredType delegating every atomic decision to is_assignable_to; a
flow-scoped NarrowEnv with branch push/pop, phi-join at merges,
whole-scope assert narrowing, declared-type anchoring (narrowing never
changes the type assignment validation uses), and fresh-environment
nested-function boundaries; guard interpretation for isinstance
(including tuple form), is None/is not None, truthiness, TypeGuard
(PEP 647, positive-only), TypeIs (PEP 742, both branches), and assert,
with loop-collected guards suppressed per TYPEINF-NARROWING-SCOPE.

Refs #317
… early-exit persistence

Statement-level flow analysis driving NarrowEnv through function bodies:
guards matched to if/assert statements by resolver-collected span; positive
frame in the if body, complement in the else, phi-join at merges; a
diverging branch (return/raise/continue/break) persists the other branch's
narrowing (the 'if x is None: return' pattern); assert narrows whole-scope
through a dedicated scope layer that never touches the declared-type anchor;
assignment narrowing via the bidirectional engine; loop bodies and
try/except walk in discarded frames; nested functions are a narrowing
boundary. End-to-end tests run parse -> resolve -> analyse_function over
the checklist's positive/complement, early-exit, loop, closure, merge, and
assignment cases.

Refs #317
… items

match subjects narrow per case (resolver Match guard matched by body span,
value patterns conservatively skipped so Never is never fabricated from
non-type pattern text). Plan checklist updated: scoped environment, guard
consumption (now including match), and declared-type-anchored assignment
narrowing are done.

Refs #317
…plied-else, issubclass/hasattr extraction

Resolver: five new NarrowingGuardKind variants extracted from if/assert
tests — issubclass, hasattr(x, 'name'), x == <literal> (case-preserving
literal text, never the lowercasing annotation parser), x in (<literals>),
and '\''key'\'' in td. Checker: equality/membership narrowing with
exact-literal complements; TypedDict key narrowing filters union members
through a NarrowContext of per-class key sets (required keys prune the
complement, optional keys survive both branches, unknown schemas stay
conservative); exhaustive fully-diverging match statements narrow the
subject afterwards by subtracting covered pattern types. issubclass and
hasattr interpret as identity until type[] modelling and synthetic-protocol
intersections exist (documented in the plan checklist).

Refs #317
…, fix too_many_lines

- narrowed_uses: tracked query running the flow walker per definition slice;
  editing one function re-executes only its narrowing (WillExecute-proven).
  Checks off the Salsa-backed use-def checklist item.
- Attribute-narrowing-across-calls tradeoff decided and documented
  (TYPEINF-NARROWING-ATTR-CALLS): survives calls by default (usable),
  narrow-attributes-across-calls = false opts into strict; parsed and
  merged in BasiliskConfig.
- Split outcome_for_kind to satisfy clippy too_many_lines after rustfmt
  reflow (the CI Lint failure on the previous push).

Refs #317
…rowing

FlowResult now carries unreachable_ranges: a branch whose guard narrows the
scrutinised variable to Never is reported unreachable — reachability derived
from the type lattice rather than pattern-matched idioms. Groundwork for the
checklist item; rules migrate off static-condition idiom matching at the
Integration stage.

Refs #317
…seline measured

Resolver extracts type(x) is C / is not C ([TYPEINF-NARROWING-TYPEOF]);
the checker narrows the positive branch by intersection and excludes C in
the negative branch only when C is @Final (NarrowContext.final_classes) —
the exact soundness split the utahplt/ifT-benchmark exercises.

ifT-benchmark harness (examples/ift_measure.rs) over a fresh clone's
Pyright suite: baseline 11/37 functions (29%) produce a flow-narrowing
signal; the silent families each map to a named pending guard form
(connectives, nesting, cross-function TypeGuard, attribute and tuple
narrowing), recorded in the plan as the ratchet floor. 8/9 flow checklist
items now checked.

Refs #317
An unannotated function's return type is now synthesized from its body
through the bidirectional engine: union of return-expression types, plus
None for bare returns or fall-through (last-statement divergence check);
no return statement at all infers None precisely. Explicitly documented as
display/assist-grade per the gradual guarantee — inferred-from-unannotated
types must never be enforced as declarations when rules consume them at
the Integration stage (differential harness stays green).

Refs #317
…ctor, method, subscript synthesis

One builtins table (bidir/builtins.rs) for builtin call returns and
str/list/dict/set method returns — the single home the checklist demands;
rule-local string tables migrate onto it at Integration. Engine synthesis
now covers: builtin calls (len -> int, sorted -> list, ...), builtin method
calls inferred from the receiver's synthesized type (dict.get ->
Optional[V]), user-class constructor calls (Named callee -> Named
instance), and subscripts (list element, dict value, tuple position via
literal index with deferred-generalization precision, str indexing, slice
preservation). Argument-dependent builtins stay Unknown - no guessing.

Refs #317
…traints and call sites

Pure module-level core (param_infer.rs) shared by the Salsa layer and the
future BSK-0001 exemption: unannotated parameters bind to input-polarity
variables; body constraints (p passed to a callee with a declared param
type) accumulate demands through the bidirectional engine, same-module
call sites contribute lower bounds, and resolution follows input polarity
(demand wins, else union of flows, else Unknown - never a guess). Engine
gains three small entry points (fresh_param_var, bind_global,
add_var_lower_bound).

Refs #317
…backdated callable interface

x = f() now resolves through the sibling's declared-or-synthesized return.
Sibling callables reach variable inference via a new callable_interface
tracked query (functions/classes only, PartialEq) so a body-only edit to a
declared-signature sibling backdates and the variable's memo survives -
the early-cutoff test caught the naive version depending on raw definition
identities and recomputing on every sibling edit. Variable-to-variable
aliases stay on the direct cycle-recovered definition_type edge.

Refs #317
imported_callable_globals maps the cross-module layer's imported_symbols
into engine scope: imported functions become gradual Callable[..., R]
from their return annotation (so calls to imported names infer returns),
classes map to their instance form, annotated variables to their
annotation, and anything without type information stays Unknown. Checks
off the same-module + imported returns checklist item.

Refs #317
…ss schemas

class_attribute_interface collects each class definition's annotated
class-level attributes (PartialEq value, backdates when schemas are
unchanged); the engine's new attribute-load synthesis answers
instance.attr from those schemas (Point().x -> float composing
constructor + schema + load). Completes the structured-AST expression
inference checklist item: constructors, attributes, subscripts,
binary/unary, conditionals, and walrus are all inferred.

Refs #317
Basilisk's pyright-compatibility layer read only a plural `stubPaths` array.
Pyright has no such key: its stub-directory setting is the singular
`stubPath`, holding one path string (its default is `./typings`).

So a real `pyrightconfig.json` -- the exact file this loader exists to
understand -- had its stub directory dropped on the floor. Nothing warned;
every stub in that directory simply failed to resolve.

The same gap existed on the `[tool.pyright]` TOML surface, and the spec had
propagated the error, listing `stubPaths` among "pyright-compatible
spellings" when pyright never accepted it.

Both spellings are now read, the explicit list winning when both appear
since it is strictly more expressive. Spec corrected to match.

Tests (verified red against the previous semantics, both reporting an empty
stub-path list):
- pyright_singular_stub_path_is_not_silently_dropped
- pyright_singular_stub_path_is_honoured_in_pyproject
- explicit_stub_paths_list_wins_over_singular_stub_path (guard; passes either
  way by design)
[STUBRES-AUTOGEN-MODES] was cited from seven places (generate/mod.rs,
runtime.rs, hybrid.rs, ast.rs, stubs_tests.rs) and [STUBRES-TYPING-STATUS]
from imports/resolve.rs:138. Neither section existed, so `grep [STUBRES-`
walked from code to nothing -- the spec-ID web had a hole exactly where the
behaviour is least obvious.

Both are documented from the implementation rather than invented:

- Generation modes: the three StubGenMode variants, which backend each uses,
  whether it needs a subprocess or source, and why hybrid is the default.
  Records that the mode is a command argument and not configuration, per
  [CHKARCH-CONFIGURATION-ONLY].
- Typing status: states the rule the resolver actually enforces -- a py.typed
  marker governs provenance and completeness classification, never
  resolution -- and why conflating them would emit "cannot be found" for a
  package that is installed and importable, suppressing the dependent cascade
  behind it.

Verified every spec ID referenced by the new prose resolves to exactly one
anchor.
LSP-ANALYSIS-MODES-SPEC said rule config consists of "severities,
per-module/per-path overrides" resolved "by walking ancestor directories and
merging cumulatively". Both halves were wrong, and it said so while linking
to the section that contradicts it.

There are no per-module or per-path override tables and no glob-keyed
sections: the model is exactly two flat maps, [tool.basilisk.rules] and
[tool.basilisk.rule-tags] ([CHKARCH-CONFIG-MODEL]). Rule entries are not
merged cumulatively either -- ancestor tables form a nearest-first chain and
the nearest table that decides a rule wins it outright. Only non-rule scalar
fields merge additively.

The distinction is not academic: a reader who believed the old sentence would
expect a distant ancestor's stricter entry to survive alongside a nearer
relaxation, and would go looking for a glob override table that has never
existed.

Swept docs/ and website/ for the same wording; the remaining occurrences
already state it correctly.
`production_manager` chose the disk cache inline, and it also constructs an
HTTPS transport, so no offline unit test could reach the choice. The result
was that `typeshed-cache-path` -- a documented config key -- had its entire
wiring untested: nothing asserted a configured directory was used, that
`typeshed-cache = false` outranked it, or that an unset path fell back to the
per-user OS cache.

Extracts the choice into `select_cache(use_cache, cache_path)` and covers all
three arms. The main test is behavioural rather than structural: it stores
through the returned cache and asserts the bytes land under the configured
directory. Asserting merely that a cache exists would still pass if the
configured path were dropped and the OS cache used instead -- which would
write outside the project while looking correct.

Verified the test kills that exact mutant: with `cache_path` ignored and
`default_cache()` returned, it fails on the missing generation directory.
The dialog scenario measured a hardcoded 120ms against two extension-host IPC
round trips plus a full DOM rebuild, on an event loop shared with every other
suite in the run. Over that budget three silent no-ops cascaded: the driver
sampled before renderPreview, .close() on a not-open dialog fired no close
event so no cancel intent was ever sent, and apply-preview hit the phase guard
and was discarded without an exception.

The driver now waits for the observable consequence at each step and reports a
failure on timeout, so a slow host fails loudly instead of asserting against a
DOM that has not reacted yet. No assertion was weakened and no product render
path was touched.

Also deletes typeshedSetting(), which read snapshot.typeshed.settings - a field
this branch removed - from zero call sites.
Source and the tests that assert it were split across the working tree;
this commits them as one coherent set so the branch builds standalone.

Typeshed acquisition:
- Send GITHUB_TOKEN/GH_TOKEN as a bearer credential, but only to
  api.github.com and codeload.github.com, matched on the parsed
  authority. A user-configured mirror never receives it. Anonymous
  GitHub allows 60 req/hr/IP, which was 403-ing CLI runs.
- Normalize a blank token to absent inside the constructor rather than
  at the environment boundary, so no caller can reintroduce a bare
  "Authorization: Bearer " that 401s a request which would otherwise
  have succeeded anonymously.
- Split cache reuse by reference kind: an exact pin reuses bytes
  regardless of age (content-addressed, re-hashed against the recorded
  SHA-256 on every load), while the moving `latest` reference keeps the
  24h TTL. Expiring a pin bought no integrity and broke offline and
  rate-limited typechecking.
- Repin typeshed-commit to the bundled SHA so the exact-pin path is
  satisfied by the embedded bundle instead of a live network call.

Mutation gate:
- Raise the per-mutant test timeout from 120s to 400s. A mutated lib
  relinks every test binary in each parallel job; that link cost alone
  exhausted the 120s budget, so killable mutants were recorded as
  TIMEOUT and credited as kills without ever being evaluated. Proven by
  running the full 29-binary set at --timeout 400 --jobs 1: 9/9 caught.
- Run mutation_kill_tests first, since cargo test stops at the first
  failing binary. Worth 31 -> 30 timeouts on its own; kept because it is
  free, not because it was the fix.

Tests and docs updated in step: credential-boundary tests asserting the
real outgoing header, pinned-vs-fresh cache reuse tests, the
source-text policy guard, and the doc-contract test covering the TTL
prose.
The analysisMode e2e drives cfg.update(..., ConfigurationTarget.Workspace),
which VS Code persists into the fixture's own .vscode/settings.json, and it
writes ofo_no_scan_target.py into the workspace root. Both are undone in the
test's finally block, so a run that dies before that block leaves the fixture
mutated - and a blanket 'git add' then commits it.

That is what shipped: settings.json pinned analysisMode to openFilesOnly, so
'basilisk.analysisMode setting has correct default' read openFilesOnly and
failed against the wholeModule default package.json actually declares.

Restores the fixture, untracks the generated Python file, and gitignores it so
the next interrupted run cannot resurrect the failure.
The exclude e2e drained the startup scan with a 500ms per-read timeout and
treated the first quiet gap as end-of-scan. That encodes an assumption about
machine speed rather than an observable fact: the server boots, resolves its
stub snapshot and indexes the workspace before anything reaches the wire, and
on a cold runner that takes far longer than 500ms.

It failed exactly where the assumption breaks. Alphabetically this test runs
before its complement, so it pays the cold-start cost the second one never
sees - CI reported 'saw []', with not even the always-scanned file publishing,
while both passed locally against a warm server.

The drain now waits up to a minute for a sentinel the configuration under test
must always scan, and only once that has arrived does it treat silence as the
answer. That ordering is what makes the absence assertions mean anything:
'excluded' and 'not published yet' are both silence on the wire, and no fixed
delay can tell them apart.

Assertions are unchanged - the pair still fails if the scanner reapplies
DEFAULT_EXCLUDES on top of a configured exclude.
`.cargo/mutants.toml` still justified `--timeout 120` as MEASURED with "~4.8x
headroom" while the Makefile passes `--timeout 400`, and credited test-package
scoping with fixing the timeouts. Both were wrong:

- The 25s worst-case figure is serial and warm. It never described the
  per-mutant test phase under `--jobs 4`, where each job works in its own
  copied tree and a mutated lib relinks all 29 test binaries per job. That
  link cost, not test execution, is what the ceiling must cover.
- Scoping the test package reduced that cost but did not remove it: 30
  timeouts survived the scoping. Raising the ceiling is what cleared them.

Also refreshes the committed report, which was the artifact of the run that
FAILED the ratchet (107 caught / 30 timeout). It now reflects a passing run:
144 mutants, 137 caught, 0 missed, 0 timeout, 7 unviable.

The baseline in mutation_scores.json is deliberately NOT tightened here. This
run is a 14-core machine at --jobs 4; the gate is enforced on CI runners with
different core counts and contention. Ratcheting the ceiling to 0 on numbers
the enforcing environment has not produced would convert a fix into a failure.
The baseline moves when CI's own merged tally supports it.
The job was killed at 20 minutes with every test already green - 30 seconds
into the final step, the PEP conformance gate. Nothing regressed; the branch
simply grew past the budget. Earlier runs never exposed it because they failed
around 15 minutes and never reached the expensive tail.

Where the 20 minutes went: the instrumented ci-profile build takes 8m20s and
the clean --release build the gate requires takes another 4m43s, so 13 minutes
is compilation before a single assertion runs. Tests then take ~4m and the
coverage pass under 1m.

A budget that stops the prime-directive gate from running is worse than no
budget, because 'the gate failed' and 'the gate never ran' look identical from
the outside. 35 leaves headroom without hiding a real hang.
The 2026-07-17 baseline recorded 105 caught and 29 timeouts at 100% kill
rate. That rate was flattering: a Timeout is folded into the killed total,
so 29 of those "kills" were mutants the suite never actually evaluated —
they expired on the per-mutant relink cost, not on hung code.

With `--timeout 400` the same pool resolves honestly. CI's merged shard
tally and a local full run agree exactly:

    total=144 caught=137 missed=0 timeout=0 unviable=7

Every axis moves the right way against 141/105/29/0: the pool grew, caught
rose by 32, missed held at zero, and the timeout credit is gone. The kill
rate is unchanged at 100% because it was always reported as 100% — what
changed is that all 137 kills are now real assertions.

`timeout: 0` is deliberate and leaves no headroom. `.cargo/mutants.toml`
states the invariant: no timeout may occur from slowness, and any that
remains is a hung-code kill. A future timeout is therefore a defect to fix,
not a budget to spend — recording anything above zero would re-open the
exact laundering channel this change closed.
configuration-editor.ts was 527 lines, breaking both the repository's
500-LOC ceiling and [VSIX-CONFIGURATION-EDITOR-FILES]'s explicit claim
that every file in that group is under it. Move the LSP seam — the
capability probe, ConfigurationEditorTransport, the six request-method
constants and workspace-root selection — into
configuration-editor-transport.ts (452 + 104 lines), re-exported from the
host module so no importer changes.

Three documented claims did not match the code:

- CLAUDE.md and [CHKARCH-PERF-PARALLEL] described "file-level parallelism
  using Rayon (work-stealing)". No Basilisk crate calls Rayon: there are
  zero par_iter/rayon:: uses across crates/, and rayon reaches the
  lockfile only transitively via salsa and ruff_db. Analysis is
  single-threaded on one large-stack thread ([LSPARCH-ARCH-STACK]), with
  Tokio concurrency confined to the LSP server. Both now say so, and note
  that file-level parallelism is unimplemented and load-bearing for no
  benchmark number.

- VSIX-SPEC claimed "every memory command is gated on basilisk.debugging".
  Seven of nine are. memoryTrackProcess is "when": false, and
  trackMemoryCurrentFile is deliberately ungated because it starts the
  session — gating it would make it unreachable.

- store-types.ts carried no Implements header; it now references
  [VSIX-ARCHITECTURE] like store.ts.

Also record why the VSIX fixture workspace carries no basilisk.enabled
key: it defaults to true, and type-checking-toggle.test.ts restores it
with cfg.update('enabled', original) — VS Code deletes a workspace
setting written back to its default, so a committed "basilisk.enabled":
true is stripped by every full run and leaves the tree dirty. Absent is
the stable state and means the same thing.
Wiring tests/basilisk and tests/dap into scripts/test-nvim.sh made 15
spec files run for the first time, and CI immediately found two real
problems that nothing had ever been in a position to catch.

Command-description assertions read the wrong field. nvim_get_commands
reports a Lua-callback command's `desc` differently across the versions
the matrix spans: on 0.11/0.12 `definition` carries the description and
`desc` is nil; on 0.13-dev `definition` is "" and `desc` carries it.
Four assertions across commands_spec, memory_spec and profiling_spec read
only `definition`, so on the nightly leg they were not checking the
description at all — they compared "" against a non-empty assertion and
failed every command, each of which has a perfectly good desc. Verified
against a real nightly build (v0.13.0-dev-1076+g69e1321731), not assumed.

The fix reads whichever field the running Neovim populates, via one
shared tests/command_desc.lua so the quirk is stated once rather than
copied four times. The blanket check also now asserts it examined at
least one command: its prefix match could stop matching and the loop
would pass vacuously, silently deleting the check.

The DAP specs need debugpy, and the test-nvim job never installed it —
only test-vscode did. Every tests/dap spec failed on CI with "debugpy
not found" while passing locally. Add debugpy==1.8.14 to that job,
matching the pin test-vscode and the devcontainer already use, and make
test-nvim.sh fail fast with the install command when it is missing
instead of surfacing ~20 unrelated-looking assertion failures.

Verified: all 12 unit spec files (385 tests) and all 3 DAP spec files
(26 tests) pass on both Neovim 0.13.0-dev and 0.12.3, zero failures.
@MelbourneDeveloper
MelbourneDeveloper merged commit a84fe91 into main Jul 21, 2026
24 checks passed
@MelbourneDeveloper
MelbourneDeveloper deleted the typeshed branch July 21, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants