[bug] eval CLI catches only RuntimeError, so unknown --transforms / bad --rank-m / --sweep dump an uncaught traceback (the #199/#201 gap, in `python -m eval`)

[real-venus]

Summary

eval/cli.py:main wraps evaluate() in except RuntimeError only. Every other CLI in the repo catches the full user-input error family and exits cleanly with code 2:

• strategy/cli.py:86 → except (ValueError, RuntimeError, MemoryError, KeyError) — the KeyError was added specifically for the unknown---transform case ([bug] strategy CLI: unknown --transform raises uncaught KeyError instead of clean exit 2 #199)
• matmul/cli.py:54 → except (ValueError, RuntimeError, MemoryError)
• attention/benchmark.py:312 → except (ValueError, RuntimeError, MemoryError) ([bug] attention.benchmark: invalid args traceback instead of clean exit 2 #201)

The eval CLI is the outlier and never received that fix.

Details

--transforms is a free-form comma string with no argparse choices (eval/cli.py:36). On a GPU machine the run gets past Backend() and reaches the smart loop, where the error escapes the except RuntimeError:

# eval/cli.py
try:
    out = evaluate(ev)
    if a.sweep:
        ns = [int(x) for x in a.sweep.split(",")]
        out["scaling"] = estimate_scaling(ns, ev)
except RuntimeError as e:          # <-- ValueError / KeyError escape
    print(f"error: {e}", file=sys.stderr)
    return 2

Uncaught paths (raw traceback + exit 1 instead of error: ... + exit 2):

• unknown --transforms → KeyError from get_transform() (strategy/transforms.py:187)
• --rank-m 0 or --rank-m > n → ValueError from multiply_subspace (strategy/subspace.py:214)
• bad --sweep foo → ValueError from int(...)

Repro (on the reference GPU box)

python -m eval --transforms bogus     # KeyError traceback, not "error: ..."
python -m eval --rank-m 0             # ValueError traceback
python -m eval --sweep foo            # ValueError traceback

Fix

Widen the handler to match the sibling CLIs:

except (ValueError, RuntimeError, MemoryError, KeyError) as e:
    print(f"error: {e}", file=sys.stderr)
    return 2

This is the exact class of bug already fixed for the strategy CLI (#199) and the attention CLI (#201), just never applied to python -m eval.

[webbrain-one]

Hi there! 👋 I’ve opened PR #346 to address this. It updates eval/cli.py to catch ValueError, KeyError, and MemoryError alongside RuntimeError, matching the error handling in the rest of the codebase. This ensures that invalid arguments (like unknown --transforms, --rank-m, or --sweep) return a clean error: ... message and exit code 2, instead of dumping an uncaught traceback. Thanks for the thorough report!