Parallelize the verify gate across modules and run the test suite only once - #14
Merged
Conversation
Every Go module loop ran serially, leaving all but one core idle. Warm verify took 86s on a sixteen-core machine. Fan for_each_go out over xargs -P, and lift the per-module coverage gate into tools/gate-coverage so its awk survives the xargs quoting. golangci-lint takes a global lock, so lint-go now passes --allow-parallel-runners; a per-module GOLANGCI_LINT_CACHE does not lift that lock. The gate now prints on failure what it used to discard, and stays silent otherwise. The e2e, Python and markdown loops stay serial. The -count=1 on the coverage runs guards golang/go#74873 and carries a comment saying so, approved for this change against the comment rule: the flag reads as cargo cult without it and was already proposed for removal once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verify depended on both test and cover-check. cover-check runs the same suite under the same -race flag, plus coverage instrumentation, so the suite ran start to finish twice on every gate. Drop test from the verify chain. The target stays for the inner loop, where an uninstrumented run is cheaper and its results cache. This needs the failure output that cover-check gained when the gate moved into tools/gate-coverage, so it cannot land before that change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11
Turned out that build/compile cache already gives us diff-scoping. Time comes mostly from sequential steps execution so now each Go module step verifies multiple modules in parallel.
-count=1argument used by coverage checking step found to be expensive too but its removal blocked by a bug in Go which is milestoned for Go 1.27 at the time of writing.