sm_110 (Jetson Thor) ctest baseline at 2daa3287f: 15 red of 485, gated on (name, mode) pairs rather than a count

[localai-bot]

Thor (192.168.68.23, Jetson Thor, sm_110) is a provisioned CUDA build-and-test host — recipe in .agents/environment.md, landed for dots3-note W0.5 (#699). This issue is the sm_110 counterpart of #907: it records what is RED there so a later row can tell a regression from the floor.

Restated 2026-08-15 at 2daa3287f. The first version of this issue described 5a0ffe9e3 (484 tests / 14 red). That baseline is superseded, and the way it was superseded is the point — see "the baseline moved" below.

Baseline

ctest -j4 --timeout 1800 at 2daa3287f, 103.35 s: 485 tests — 468 passed | 2 skipped | 15 FAILED. Skips are test_modelopt_mixed_precision_checkpoint and test_voxtral_e2e, both for an absent checkpoint. Still current at f365cc299 (nothing moved in src/, include/, tests/, CMakeLists.txt).

All 15 reproduce at -j1 with identical failure modes — 15/15 in 18.59 s, artifact /home/mudler/thor-w05/ctest-j1-rerun.log on the box. None is a ctest -j starvation artefact.

Test	Mode	Cause
test_platform	Failed	:307 asserts is_device_capability_family(120); Thor is 11.0. The TEST hardcodes GB10
test_linear_method	Failed	:246 after == before + 1 → 0 == 1, "MXFP4 fused gate_up … fused path ran". Also red on GB10 (#907)
test_qwen3_5_gdn_spec_routing	SEGFAULT	#960. Also red on GB10 for a different reason (#907) — expect Failed, not green, after #960
test_qwen3_5_gdn_spec_routing_glue_fuse_off	SEGFAULT	#960, same #907 caveat
test_qwen3_5_gdn_spec_routing_fused_chain_off	SEGFAULT	#960, same #907 caveat
test_deepseek_v2_forward	Failed	:559 built without the vendored FlashAttention-2
test_capi	SEGFAULT	:487. Pre-existing and arch-independent; same crash on GB10 (#907)
test_ops_fp8_cutlass	SEGFAULT	#960
test_ops_fp8_cpu	SEGFAULT	#960, at :279 "G2: CPU QuantFp8Static equals CUDA QuantFp8Static byte for byte"
test_ops_moe_grouped	Failed	#962 — NVFP4 block8-vs-block16 M=8 K=4096 N=4096 bitdiff=15/32768 on a kernel that IS enabled for [110]
test_ops_fused_chain	SEGFAULT	#960
test_ops_mla_prefill	Failed	:340, :437 FA-2 absent
test_ops_mla_chunked_context	Failed	:790 FA-2 absent
test_mla_attention_block	Failed	:999, :1044 FA-2 absent
test_op_parity	Failed	:2487 output_cbor_sha256 mismatch in a case named dgx-only that runs anyway

Four causes: six are #960, four are the missing vendored FA-2, two hardcode GB10, and three stand alone (test_capi and test_linear_method, both already red on GB10, plus #962).

The gate: (name, mode) pairs, not a count of names

A row regresses on Thor when it adds a NAME or changes a recorded MODE. Counting names is provably insufficient, and this issue's own history is the proof: between 5a0ffe9e3 and 2daa3287f, five tests went Failed → SEGFAULT with no name change, and the list grew only because the same upstream change happened to also ship a new test file. A name-counting gate would have scored #960 — five crashes it introduced — as GREEN. That is why the Mode column above is part of the baseline and not decoration.

The baseline moved, and that is how #960 was found

At 5a0ffe9e3 the FP8 group failed by throwing vt: no kernel for op QuantFp8Static (id 52) on device cuda — a loud, correct refusal. At 2daa3287f the same request silently takes the portable CPU reference tier and segfaults. cutlass-fp8 is ENABLED on GB10 and DISABLED for [110], so the crashing path is unreachable on the gate host and no CI lane could see it. Detail and attribution in #960.

Re-measure this baseline whenever the base SHA moves across src/, include/, tests/ or CMakeLists.txt. A stale baseline is worse than none, because the next agent reads a regression as the floor.

Two things that are NOT in the 15

test_engine_core_proc — the documented parallel-ctest timing flake — failed a third -j4 run while passing both baseline runs. It is not part of the baseline; re-run it alone before believing it.

test_serve_low_tools fails with FileNotFoundError: 'shellcheck' if the container image lacks shellcheck. That is the harness defect in #961, not an sm_110 fact; the recorded Dockerfile installs shellcheck so the baseline is 15 and not 16.

No fix is attempted here. Three of the four causes are the build honestly refusing a feature the arch does not have; the substantive one is #962.

[localai-bot]

Splitting the one substantive finding out of this baseline, so it does not get closed wholesale with the known reds.

#962 now owns the test_ops_moe_grouped block8-vs-block16 bitdiff=15/32768. It is the only one of the 14 that is not feature absence: marlin-nvfp4 is ENABLED for [110], so that is a live kernel disagreeing with itself, not a gap. The reasoning was already in this issue's text, but a baseline issue is exactly the kind of record that gets closed as "known reds on this host", and this one should outlive that.

Also related, found in the same bring-up: #960 (FP8 on a CUDA device without cutlass-fp8 now SEGFAULTs rather than refusing loudly, since #842) and #961 (test_online_gate_startup.py:259 guards shellcheck absence with a check that cannot fire, so a missing tool reads as a code failure).

Please keep this issue for the remaining 13 and the differential gate.