[Bug] AOT Selftests intermittently exits 1 with no failing test and no TAP trailer — a host crash distinct from #988, and misattributed to it

[azchohfi]

Summary

The AOT Selftests leg intermittently exits 1 with no failing test, no TAP trailer, and no watchdog message. It reproduces on clean main and is distinct from both #988 (watchdog timeout) and #1037 (*_Removed assertion cluster).

Because #988 is the available explanation for "red AOT leg with no obvious cause", this shape gets triaged into it and disappears. The TAP trailer is the only reliable discriminator.

Signature

	not ok	# Total failures trailer	timed out … in flight	orphan dotnet at teardown
#988 watchdog	present	present	present	6
#1037 *_Removed	present	present	–	6
this crash	0	0	0	6

Orphaned processes are not a discriminator

Worth stating explicitly, because it reads like crash evidence: Terminate orphan process: pid (…) (dotnet) appears 6 times in every failure of every shape, including plain assertion failures. It is a teardown artifact of any failed job, not a crash indicator. Only the trailer separates the shapes.

Measured rate

Sampled every AOT Selftests job execution across recent ci.yml runs, filter=all so re-run-masked first attempts are visible:

branch	executions	failures	rate
main	101	4	3.96%
a feature branch (PR #1011)	48	2	4.17%

Of main's 4, three carry a trailer (#988 / #1037 family) and one is this crash: 3d0a94ad, run 29319850743 — not ok=0, trailer=0, timeout=0.

Why the obvious control run cannot settle it

At the measured ~4% rate, a single clean re-run on main passes 95.8% of the time even when the defect is fully present. One control run cannot come out the other way in any useful sense, and ~70 executions are needed before a zero-failure result is 95%-confident evidence of a different rate.

This bit me directly while investigating: an initial 25-run main sample showed 0 failures and pointed at "the feature branch causes it". Widening to 101 executions surfaced 4 and inverted the conclusion. At 25 draws, a 4% defect shows zero 34% of the time — the sample size, not the code, was deciding the answer.

Repro / triage

# every AOT Selftests execution, including re-run-masked first attempts
gh api "repos/microsoft/microsoft-ui-reactor/actions/runs/<id>/jobs?per_page=100&filter=all"

# classify: trailer present => #988/#1037 family; absent with exit 1 => this bug
gh api "repos/microsoft/microsoft-ui-reactor/actions/jobs/<job>/logs" | grep -c "# Total failures"

filter=all matters — the default returns the latest attempt only, so any failure that passed on re-run is invisible. That default makes the standard query structurally unable to observe this bug.

Suggested next step

Have the AOT selftest host emit a trailer unconditionally (even on abnormal termination), so "no trailer" becomes an explicit, greppable host died signal rather than an absence that has to be inferred. Today the absence of output is doing the work of a diagnostic, which is why the shape is invisible to triage.

Found while investigating CI on PR #1011 (issue #989).