The aid pty tests flake on a loaded cargo test --workspace run

[blooop]

Finding

rust/aid/tests/interactive.rs fails intermittently on a loaded cargo test --workspace, and passes reliably when the suite is run alone.

Measured on this machine (devcontainer, pixi run cargo test --workspace, main at 9392779):

• suite alone, cargo test -p aid --test interactive: 3/3 green
• full workspace run: 2 failures in 4 runs

The failure seen was a_pasted_multi_line_prompt_arrives_whole_rather_than_leaking (aid/tests/interactive.rs:315), with "press Enter" never appeared after the 60s deadline in wait_for (:205).

The tests drive the real binary through a pty and poll wait_for at 25ms against a 60s deadline, so they are sensitive to scheduling pressure from the rest of the workspace run. Each also spawns python3 launch_scenario.py to build a scratch world first.

Why it matters beyond the noise

CI green becomes probabilistic, and a real regression in the interactive flow arrives looking exactly like the flake — which is the state in which flakes stop being investigated.

Shape

Unclear, deliberately. Worth establishing first whether the deadline is simply too tight under load, or whether there is a genuine lost-wakeup between the pty reader thread (:148) and wait_for. Options in rough order of cost: raise the deadline, serialise the pty tests against each other, or hand the reader thread a condvar instead of polling.

Found while adding a pty regression test in the DEVLAUNCH_NO_TTY fix; that test is not the cause — the baseline flakes more often without it.

[blooop]

Diagnosed and fixed in #414 (open, not merged).

Mechanism, confirmed. Not a lost wakeup, not a dead reader thread, not a deadline that is too tight. PtyAid::send_line wrote the typed line and its Enter as two write_all calls on the pty master. A master write reaches the line discipline in one go, so two writes are two of those with a gap between them: the first completes fix this and wakes aid's read, read_terminal_submission drains what the terminal holds right now (its zero-timeout poll is deliberate), and while the second write is still in flight the paste's tail is not yet a completed line, so the prompt arrives cut at line one. The test's own comment claimed "One write, as a terminal delivers a paste"; the helper it called had never made that true. One write_all of line plus newline, and every completed line of a paste becomes readable at the same instant.

Confirmed by knob rather than by argument: inserting a deliberate 20ms sleep between the two writes moved the failure rate from 14 in 100 to 19 in 20.

Numbers. Suite binary run back to back on 8 cores under a load average of 12 to 25:

• before: 86 green in 100, all 14 failures at interactive.rs:291 (paste truncated to line one)
• after: 100 green in 100

The reported symptom did not reproduce. "press Enter" never appeared after the 60s deadline was not seen once in 280 runs of this suite or in any of the 6 full-workspace runs measured. The deadline in wait_for is left alone: raising it for a mechanism nothing confirmed would buy only a slower failure. Every failure this ticket did reproduce was the paste-tail assertion, in the test the ticket names.

Other flakes seen while measuring, not this ticket's and not touched: flows::launch::tests::a_contended_up_of_a_running_workspace_runs_no_up_at_all, timing::tests::stages_closed_out_of_order_do_not_stay_on_the_clock (twice), and dl/tests/interrupt.rs's closing_the_terminal_mid_up_removes_the_token_file_and_kills_the_up. Also: a_ctrl_c_at_the_editor_tears_the_whole_boot_down hangs its "the orphaned devpod up must have been killed" assertion on a wait_for whose subject is a sleep 30, so a kill that never happened still satisfies it 30 seconds later. Separate weakness, separate ticket.

[blooop]

breadcrumb: mechanism confirmed and fixed in PR #414 (86/100 → 100/100). Four further flaky tests and one vacuous assertion found while measuring, carried to #416 rather than folded in — and the symptom this ticket originally recorded ("press Enter" never appeared) did not reproduce in 280 runs, which #416 also carries.

[blooop]

breadcrumb: wf-review on #414 — approve. Mechanism reproduced independently (two writes + 20ms gap fails every time; base helper 2 failures in 60 on an idle box, this PR 0 in 40), and the test still grips production: disabling the drain at dl/src/lib.rs:287 fails it with exactly the truncated paste. Note for whoever closes this ticket: the symptom recorded here ("press Enter" never appeared) is not what was fixed — close against #414 and #416. #414

[blooop]

Closing against PR #414 and #416, per the review's instruction. #414 fixed the one flake this ticket names at its mechanism (the paste arrived as two pty writes; it is now one write_all), measured 86/100 before and 100/100 after, knob-confirmed. The symptom string recorded here ("press Enter" never appearing) never reproduced in 280 runs; it and the four other flakes found while measuring are carried by #416.