Skip to content

ci: api-diff-closure 2-minute packages.Load budget fails on loaded runners #2341

Description

@yuanchen8911

Summary

TestRunReportsCurrentAliasMappings runs packages.Load over the entire repository under a hard 2-minute budget. That budget is roughly 3x the local runtime but marginal on CI hardware, so the test fails intermittently on loaded runners and blocks unrelated PRs.

Evidence

Context Result
Local, warm cache passes in 43s
CI, PR #2336 run 1 FAIL at 120.13s
CI, PR #2336 run 2 FAIL at 120.03s
CI, PR #2336 run 3 pass
CI, PR #2321 (earlier same day) same failure, cleared on retry

Failure signature is identical every time:

--- FAIL: TestRunReportsCurrentAliasMappings (120.13s)
    main_test.go:292: run() error = load packages: context deadline exceeded
FAIL	github.com/NVIDIA/aicr/tools/api-diff-closure	120.274s

The 120.x wall is the timeout itself, not a hang: goListTimeout = 2 * time.Minute (tools/api-diff-closure/main.go:38), applied at main.go:139.

Impact

  • Blocks merges on PRs that have nothing to do with api-diff-closure. docs(defaults): correct MirrorDefaultKubeVersion rationale comment #2336 is a single Go comment line and still needed three runs.
  • Trains maintainers to reflex-retry a red tests / Test, which erodes the signal — a real regression looks the same as this on first glance.
  • Costs full CI runs: each retry re-runs the whole tests / Test job (~9 minutes).

Why it is timing-sensitive

The test type-checks every package in the module from the repo root. Runtime scales with module size and depends heavily on build-cache warmth, so a PR touching a widely-imported package (e.g. pkg/defaults, imported almost everywhere) invalidates much of the cache and pushes the load toward the ceiling. Module size only grows, so this will get worse.

Options

  1. Raise goListTimeout — smallest change. 5 minutes would give ~7x headroom over the observed local runtime. Treats the symptom but is cheap and immediate.
  2. Narrow the load scope — the test only needs the alias targets under pkg/client/v1 and their transitive deps, not the whole module. Biggest win if feasible.
  3. Warm the build cache in CI before this job, so cold-cache runs stop being the worst case.

Option 1 alone would likely have prevented every failure observed so far.

Not caused by

Neither #2336 nor #2321 changed anything reachable from this test; main was green throughout.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/citheme/ci-dxCI pipelines, developer experience, and build tooling

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions