Skip to content

Unparseable files: text output drops them entirely, and a user syntax error exits 3 (Internal failure) masking exit 1 #384

Description

@MelbourneDeveloper

A file Basilisk cannot parse is handled correctly in JSON and badly everywhere
else. Three separate defects share one cause: CheckOutcome::failures is a
side-channel that only the JSON renderer knows about.

[CHKARCH-CLI-OUTPUT-FAILURES] already argues this case for JSON, and its
reasoning applies verbatim to text:

Omitting it rendered [] — byte-for-byte the answer a clean file gets — for a
file that was never checked, so every consumer reading the report rather than
the exit code was told a file with a syntax error had no problems. […] a
report that contradicts it is worse than one that says nothing.

The spec section is implemented and tested only in
basilisk-cli/src/output/json.rs.
Text has no counterpart, no spec clause, and no test.

Repro

$ mkdir /tmp/bsk_syn && cd /tmp/bsk_syn
$ printf 'x: int = "no"\n' > good.py
$ printf 'obj = object()\nobj.\n' > broken.py

1. Text output drops the unanalysable file from the report and the count

$ basilisk check .
error[assignment_compatibility]: Type mismatch: `x` is annotated `int` (int) but assigned LiteralString
 --> ./good.py:1:1
...
Found 1 diagnostic (1 error).
2026-08-01T06:52:51Z ERROR basilisk: error processing file path=./broken.py error=syntax error in ./broken.py: Expected an identifier at byte range 20..21

The report body has no entry for broken.py, and the summary says 1
diagnostic. --output json on the same run returns 2 entries. Text and JSON
disagree about what the run found.

The only signal in text mode is a tracing line on stderr, emitted after the
summary and outside the report — so it is lost to 2>/dev/null, to any log
filter, and to anyone reading the report rather than watching the console. With
no other diagnostics present, the text renderer prints nothing at all:

// crates/basilisk-cli/src/main.rs:426
let exit_code = if total == 0 && outcome.failures.is_empty() {
    println!("{}", "All checked. No issues found.".green().bold());
    0
} else if total == 0 {
    0            // <- failures present: prints nothing, reports nothing
}

That branch was clearly added to stop the "All checked. No issues found." lie —
correct instinct, but silence is the other half of the same bug.

2. A user syntax error exits 3 (Internal failure) and masks exit 1

$ basilisk check .          # 1 type error + 1 unparseable file
$ echo $?
3
$ rm broken.py && basilisk check . ; echo $?
1

[CHKARCH-CLI-EXITCODES] defines 3 as Internal failure. A syntax error in
the user's own Python is not internal to Basilisk — it is the most ordinary
finding a checker can have. Two consequences:

  • CI that branches on the documented codes ([CHKARCH-CLI-CI] tells consumers to
    do exactly that) is told the tool broke, not that the code has problems.
    Pipelines that retry or page on 3 will do so on a plain typo.
  • The failure exit overrides the diagnostic exit unconditionally
    (if outcome.failures.is_empty() { diagnostic_exit } else { 3 },
    main.rs:360), so one
    unparseable file hides the fact that real errors were found.

This reproduces on this repository: basilisk check . exits 3 because
basilisk-zed/tests/fixtures/completions.py ends in obj. — a deliberately
incomplete completion fixture. The fixture is correct and must stay invalid; the
handling of it is what is wrong.

3. The position is a byte range, not line:col

syntax error in ./broken.py: Expected an identifier at byte range 20..21

No user can act on byte 20. Every other checker reports file:2:5: invalid syntax. The offset comes straight from ruff_python_parser, which hands us a
TextRange — the line/column conversion the rest of the pipeline already does
for diagnostics is simply never applied here. The entry also carries no code and
therefore no see: URL.

Note the JSON entry anchoring at line 1, column 1 is specified behaviour and
should not change; this is about the message text.

Acceptance

  • Extend [CHKARCH-CLI-OUTPUT-FAILURES] to cover text output, and render
    unanalysable files in the text report with the same standing they have in
    JSON — visible in the body, counted in the summary, on stdout with the
    report rather than as a stderr log line.
  • Decide and specify the exit code for an unanalysable file. 1 matches its
    severity and keeps 3 meaning what the table says. If it must stay
    distinct from 1, it needs its own row in [CHKARCH-CLI-EXITCODES] — not
    a reuse of "Internal failure".
  • Stop a failure from masking a real diagnostic exit.
  • Convert the parser's byte offset to line:col in the message.
  • Regression tests for each: a text-mode assertion mirroring the existing
    cli_binary_tests.rs JSON assertions, a text-vs-JSON agreement test on one
    mixed run, and an exit-code test for failure-only and failure-plus-error
    trees.

Metadata

Metadata

Assignees

No one assigned

    Labels

    high-priorityHigh priority — schedule ahead of other feature workspec-violationCode functionality/logic does not match its spec (see SPEC-CONFORMANCE-AUDIT-PLAN.md)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions