Skip to content

4712dac40 (VT-ACT-ROUND-POLARITY) reds four suites on main: the bf16 activation narrowing exceeds error floors those suites never had re-derived #1458

Description

@localai-bot

Found while gating #1403 (PR #1457). Filed in flow; it is not that row's defect,
and it is not repaired here because the decision it needs belongs to
VT-ACT-ROUND-POLARITY.

What happens

Four suites are red on main at 0adeb8b0e, CPU-only Release, x86_64:

  • test_ltx2_text_encoder
  • test_muse_glimmer_text
  • test_muse_glimmer_text_fallback
  • test_minimax_music3_ar

test_ltx2_text_encoder states the shape plainly — two bf16 error floors, both
exceeded, both by roughly 20 to 30 percent:

tests/vllm/models/test_ltx2_text_encoder.cpp:2407: ERROR: CHECK( video_bf16 <= video_floor ) is NOT correct!
  values: CHECK( 0.1323 <= 0.109394 )
tests/vllm/models/test_ltx2_text_encoder.cpp:2409: ERROR: CHECK( audio_bf16 <= audio_floor ) is NOT correct!
  values: CHECK( 0.0752773 <= 0.0573374 )
[doctest] test cases:   27 |   26 passed | 1 failed | 0 skipped
[doctest] assertions: 4118 | 4116 passed | 2 failed |
[doctest] Status: FAILURE!

This is a deterministic assertion, not a load or memory artifact. The box was
busy when it was first seen, so it was re-run serially and it failed serially in
101 s across the four.

Attribution, proven by mutation rather than inferred

4712dac40, fix(VT-ACT-ROUND-POLARITY): narrow act(gate) to the input dtype, which upstream pins bit-exactly (#1322, PR #1347). It changes
src/vt/cpu/cpu_ops.cpp by +42/-3 and adds tests/vt/test_ops_activation.cpp.
It does not touch any of the four suites above.

Both directions were measured in one build directory, changing only that one
file, with the compile exit code taken beside every arm and sha256sum on the
file before and after:

src/vt/cpu/cpu_ops.cpp compile four suites
at 0adeb8b0e 0 4 failed
reverted to 4712dac40^ 0 4 passed, test_ltx2_text_encoder 27/27, 4118 assertions, SUCCESS!
restored, sha256 re-matched 0 4 failed

The four also passed at b537a5344, three commits earlier, in a full
ctest run of 567 tests taken before the merge, at 3 to 6 s each.

Why this is not repaired in flow

The in-flow rule covers a small and clear fix. This one is neither, because the
question it asks is a numerics decision and not a defect with an obvious repair.
Either the floors were calibrated against the rounding polarity 4712dac40
corrected, in which case they need re-deriving against the oracle and the change
is right; or the narrowing is wider than upstream's and the floors are right.
Deciding that from the outside would be guessing, and picking the first branch
by editing the four floors is exactly the "make a red gate green by widening a
scope" move AGENTS.md prohibits.

Owner: row VT-ACT-ROUND-POLARITY, spec .agents/specs/vt-act-round-polarity.md.

Reproduce

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4 --target test_ltx2_text_encoder
ctest --test-dir build -R '^test_ltx2_text_encoder$' --output-on-failure

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions