test(ci-flake): M2 — migrate call sites to the testutil gate, add deterministic httptest coverage - #593
Merged
Merged
Conversation
…erministic httptest coverage
Closes AC1, AC2, AC3'(a/b/c), AC4, AC6, AC7 of m-ci-flake-systemic-fix.
Executed by codex gpt-5.6-sol; all gates re-run by the controller OUTSIDE the
sandbox (in-sandbox verdicts are uninformative — workspace-write denies
loopback binds, which is indistinguishable from a real regression).
The central change: AILANG's live-network tests no longer run by default.
pre-sprint : --- FAIL: TestNetHttpPost/httpPost_to_httpbin.org (0.37s)
net_test.go:380: got: <html>503 Service Temporarily Unavailable
post-M2 : --- SKIP: TestNetHttpPost/httpPost_to_httpbin.org
--- PASS: TestNetHttpPost/local_success_response
--- PASS: TestNetHttpPost/local_non-2xx_response
That 503 was measured first-party on dev during this iteration, with no proxy
env set — i.e. #588/#561 reproducing live. CI was green only because the old
Getenv("CI") opt-out hid it there, which is the defect class this sprint removes.
Changes (12 test files, 8 packages, +86/-115; no production code, no workflows):
- internal/pkg/gitcache_test.go, internal/effects/net_test.go — inert gates and
the SKIP_NET_TESTS/CI/GITHUB_ACTIONS opt-out replaced by
testutil.RequiresLiveNetwork
- internal/effects/net_test.go — live non-2xx now tolerated-and-logged, and NEW
deterministic httptest coverage for netHTTPPost success + non-2xx. Requires
ctx.Net.AllowHTTP and ctx.Net.AllowLocalhost (both default false; an httptest
URL is http://127.0.0.1:<port>, blocked twice over)
- cmd/ailang/main_test.go — runCLI, buildAilang and runAilangBin routed through
testutil.RunBounded (120s cap), including the go-run helper the design doc missed
- cmd/ailang/main_run_pipe_test.go — eventOneBudget removed, minGap untouched
- internal/eval_harness/reference_solutions_test.go — per-language warm-up run
(the actual #587 fix; the 60s constant was only the symptom)
- 6 further inert gates removed, each guarded body re-read in context first
- internal/coordinator/provider_script_test.go — allowlisted with a reason: it
gates Unix shell signal semantics, not network, so the network helper would
misdescribe it
Verified outside the sandbox:
full suite unpoisoned rc=0 106 ok 0 FAIL
full suite poisoned [AC3c] rc=0 106 ok 0 FAIL (pre-sprint: rc=1, 1 FAIL)
poisoned ./internal/pkg/ rc=0 (pre-sprint: rc=1, git clone exit 128;
unpoisoned control rc=0, so the poison
is the mechanism)
AILANG_LIVE_NET=1 [AC4] --- PASS (1.55s, a real clone — not a skip)
live lane + poison [AC11] FATAL "live network lane is misconfigured"
gofmt 0 files, go vet rc=0
grep sweeps: testing.Short() 7->0 files, Getenv(CI|GHA) 2->1 (allowlisted),
eventOneBudget 4->0, minGap 3 (control, untouched),
60*time.Second 1->0, warm 0->4
Mutation-tested by the controller, negative control green first, each mutation
proven landed by sha256 + byte-diff, both reverts byte-identical:
- remove the live gate -> SKIP becomes PASS (subtest runs live again)
- remove AllowLocalhost -> both deterministic subtests FAIL
NOTE for M4/M5 wiring: AC3'(b) is NOT rc-checkable. The first mutation left the
package rc=0 because httpbin happened to answer that time. AC3'(b) must assert
on the literal "--- SKIP:" lines, never on the exit code.
Two plan discrepancies found and reported by the executor, both confirmed:
- `go build ./...` is not a usable gate here: it fails identically on unmodified
dev (cmd/wasm and gen/main have no native main). Not caused by M2.
- PR #532's sync.Once builder was ALREADY on dev via #564 (3c28cc3, merged
2026-08-03), so the "#532 collides with M2" blocker was already moot. #532
closed as superseded.
Co-Authored-By: codex <gpt-5.6-sol>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sunholo-voight-kampff
enabled auto-merge (squash)
August 5, 2026 12:23
|
sunholo-voight-kampff
added a commit
that referenced
this pull request
Aug 5, 2026
… M2 landed, and a three-iteration-old blocker was already dead Gate 4 + Gate 5 record for mission iteration 145. Pick was Mark's human directive (outranks the queue): D5 = Option A, queue B separately, iter-141 carve-out accepted. Landed: - D5 applied in full: AC3 -> AC3'(a/b/c); NEW AC10(d) asserting the residual as open, with a Proxy: http.ProxyFromEnvironment arm as its known-positive control, designed to go RED when Option B lands - Option B queued as its own NEW-DOC item (m-net-effect-proxy-boundary) - CI-flake M2 landed (368f940, PR #593), evaluator sonnet PASS 91/100 r1 - Weekly external-issue sweep run as a measurement: #588 (closed by M2 itself), #589/#590 batched into one queue row RETRO — one skill edit, at the >=2-instance bar. Gate 2 gains "the item's DECLARED BLOCKERS are claims too": re-verify a blocker's PURPOSE against HEAD rather than its state/mergeable, because a superseded PR stays OPEN and CONFLICTING forever — nobody rebases a PR whose reason is gone, so staleness looks identical to blocking. instance 1: 2026-08-04 motoko_agent batch (1 of 3 already fixed, 1 superseded) instance 2: #532, superseded by #564/3c28cc322 on 08-03 while iterations 142/143/144 planned around it and 145 spent a controller decision on it. I then asserted its fix was "still wanted" without measuring HEAD — rule 3b(v)(b) exactly. Watch-item (instance 1, bar is two): a search pattern can match too MUCH. My wait condition grep -q "poisoned rc=" matched the superstring "unpoisoned rc=0" and reported a 106-package suite finishing with 0 packages. Rule 3a covers empty results; nothing covers a silent adjacent-superstring match. Ruled out: iteration 144's "$MODEL arrived empty, check the driver's export" is REFUTED. MODEL is never exported (grep 0; control: role vars export = 1); the driver passes it as a `claude --model` CLI argument. That fix would have done nothing — iter-144 ran on Fable because the ordered preference probe fell through, not because of an export bug. Deferred with measured numbers (Standing rule 2, not fixed here): class C2's generator survives — 31 absolute context.WithTimeout(Background(), N) sites vs 2 deadline-derived, gatelint R1-R3 cover none. serve_api_mcp_surface_test.go (30s budget, 10.34s actual, 2.9x margin) was dormant until M2 removed its inert gate; watch it on the first dev run after M4. metered=$0.00 of the $5 ceiling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone M2 of
m-ci-flake-systemic-fix. Closes AC1, AC2, AC3′(a/b/c), AC4, AC6, AC7.Unblocked by Mark's
D5= Option A decision (2026-08-05): AILANG'sNeteffect stays outside the poisoned-proxy egress boundary, AC3 is narrowed to where the poison actually governs, and the residual is asserted openly.The central change
AILANG's live-network tests no longer run by default.
That 503 was measured first-party on
devduring this iteration with no proxy env set —#588/#561reproducing live. CI was green only because the oldGetenv("CI")opt-out hid it there, which is exactly the defect class this sprint removes.Verified outside the sandbox
In-sandbox verdicts are uninformative (
workspace-writedenies loopback binds, indistinguishable from a real regression), so the controller re-ran every gate:./internal/pkg/(AC3′a)git clone exit 128; unpoisoned control rc=0 → the poison is the mechanism)AILANG_LIVE_NET=1(AC4)--- PASSin 1.55s — a real clone, not a skiplive network lane is misconfiguredGate is falsifiable in all three directions: default→SKIP, live→PASS, live+poison→FATAL.
Mutation-tested
Negative control green first; each mutation proven landed by sha256 + byte-diff; both reverts byte-identical.
SKIPbecomesPASS(runs live again)AllowLocalhost→ both deterministic subtests FAIL (so the new coverage is real, not vacuous)⚠ For M4/M5 wiring: AC3′(b) is not rc-checkable. The first mutation left the package at rc=0 because httpbin happened to answer that time. It must assert on the literal
--- SKIP:lines.Two plan discrepancies found, both confirmed
go build ./...is not a usable gate here — it fails identically on unmodified dev (cmd/wasm,gen/mainhave no nativemain).#532'ssync.Oncebuilder was already on dev via#564(3c28cc322, merged 2026-08-03), so the "fix(ci): one binary build, not fourteen — cmd/ailang had grown into its Windows timeout #532 collides with M2" blocker was already moot.#532closed as superseded.Executor: codex
gpt-5.6-sol. Evaluator: sonnet.🤖 Generated with Claude Code