Skip to content

ci(gate): cut PR validation cost — cached test fixtures, correct worker count, narrower UI scope - #2426

Merged
JSONbored merged 9 commits into
mainfrom
claude/speed-up-test-d1-fixture
Jul 2, 2026
Merged

ci(gate): cut PR validation cost — cached test fixtures, correct worker count, narrower UI scope#2426
JSONbored merged 9 commits into
mainfrom
claude/speed-up-test-d1-fixture

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes found while profiling why validate-code was taking 6-10 minutes per PR on the self-hosted runner (real step timings pulled from a recent run: Test with coverage alone was 5m39s of a 7m52s job):

  • test/helpers/d1.ts: TestD1Database's constructor rebuilt the whole schema from scratch on every instantiation (~1500 call sites across the suite), reading and executing all migrations/*.sql files sequentially. Cache the concatenated migration SQL once per worker process instead of re-listing and re-reading ~90 files on every call. This still calls .exec() per instance (unavoidable on this repo's pinned Node 22 -- see below), so it removes only the redundant filesystem work, not the DDL execution itself.
  • .github/workflows/ci.yml — worker count: Test with coverage used --maxWorkers=100%. Node's os.cpus() reports the HOST's full core count inside a container, ignoring the Docker cgroup CPU quota — confirmed directly on the runner (docker exec into it reports 8 cores while its actual Docker CPU limit was 2, since raised to 4). This meant vitest was very likely oversubscribing its worker pool well past what the runner could actually execute in parallel. Pinned to --maxWorkers=4 to match the runner's real quota.
  • .github/workflows/ci.yml — UI path scope: the ui changes filter included src/**, so any backend-only PR ran the full ui:lint/ui:typecheck/ui:test/ui:build toolchain even though it couldn't possibly affect the UI. Split into ui (the UI app/extension's own code, plus dependency bumps — still triggers the full toolchain) and a new uiContract category (src/**, the openapi-writer script — triggers only the lightweight ui:openapi:check drift check). UI/MCP version audit is re-scoped to ui || mcp since it only checks an MCP package-version string and never reads the OpenAPI contract.
  • .github/workflows/ci.yml — MCP CLI-harness scope: test/unit/mcp-cli-*.test.ts and mcp-discovery.test.ts (6 files, subprocess-per-test for the CLI-harness files, a real subprocess-spawned MCP protocol handshake for mcp-discovery) were unconditionally bundled into Test with coverage regardless of whether anything relevant to them changed. Verified by direct-import inspection that all 6 are genuinely self-contained w.r.t. root src/** — they only import node:* builtins, vitest, and their own test harness, and spawn the built packages/gittensory-mcp CLI as a real subprocess. This mirrors the existing mcp filter's own established trust boundary (a self-contained package build). Added an mcpCliHarness filter scoped to exactly those 6 files + packages/gittensory-mcp/**, and skip them via vitest --exclude when it's false on a PR. test/unit/mcp-output-schemas.test.ts is deliberately not included in this filter or the exclude list: it imports src/mcp/server.ts in-process, and server.ts alone directly imports ~40 other src/ modules, so its real dependency surface is practically all of backend — it always runs whenever backend does. Push events always run the full suite unchanged.
  • package.json: added test:changed (vitest run --changed=origin/main) for the local edit-test-fix loop — real import-graph-aware test selection, not just path globs. Not wired into CI or the Codecov gate; the authoritative pre-push check stays test:coverage/test:ci.

Also raised the 3 gittensory-labeled self-hosted runner containers' Docker CPU limit from 2.0 to 4.0 (the VPS has 8 total vCPUs), applied live via docker update --cpus (no container restart, no disruption to in-flight jobs) and persisted in docker-compose.override.yml on the host — not part of this diff since that file isn't committed to the repo.

Update: two follow-up fixes after the first push

  1. CI was actually red (Test with coverage failing): the original test/helpers/d1.ts cached a DatabaseSync.serialize()/deserialize() snapshot to clone per instance. Those methods don't exist on Node 22 at all (confirmed absent on 22.23.1, present only from Node 24+) — this repo's .nvmrc pins Node 22, so every test that touches TestD1Database crashed in CI (db.deserialize is not a function) while passing locally on a newer Node. Replaced with a Node-22-safe cache of the concatenated migration SQL (see the test/helpers/d1.ts bullet above) — verified green on Node 22.23.1 with identical coverage numbers to the pre-fix run.
  2. AI review blocker (reachable CI blind spot): the original mcpCliCluster filter hand-picked a handful of src/** files it assumed were mcp-output-schemas.test.ts's real dependency surface. That guess missed dozens of src/mcp/server.ts's actual direct imports (src/github/app.ts, src/signals/slop.ts, src/settings/autonomy.ts, etc.), so a backend change outside the guessed list could skip that test with no rerun to catch a regression. Fixed by narrowing the skippable set to only the 6 files verified genuinely self-contained (see the mcpCliHarness bullet above); mcp-output-schemas.test.ts is no longer excludable under any condition.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused: test-infra + CI-workflow only, no product code changes.
  • This follows CONTRIBUTING.md.
  • No issue is linked — this is CI/test-infra maintenance found via direct profiling, not a pre-filed bug.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck (verified on Node 22.23.1, matching CI's pinned .nvmrc)
  • npx vitest run test/unit/workflow-runner-labels.test.ts test/unit/codecov-policy.test.ts test/unit/change-guardrail.test.ts test/unit/agent-guardrail-paths.test.ts test/unit/path-matchers.test.ts (every test that reads ci.yml's content) — all pass unchanged.
  • Full npm run test:ci — run on both Node 22.23.1 (matching CI's pinned .nvmrc) and a newer local Node, all green, identical coverage numbers on both.
  • npm run test:workers — passes on Node 22.23.1.
  • Manually verified the mcpCliHarness --exclude/skip logic locally (both the skip and no-skip branches): with the 6 harness files excluded, only mcp-output-schemas.test.ts remains and passes; with nothing excluded, all 3 sampled files pass.
  • npm audit --audit-level=moderate: 0 vulnerabilities.

Notes

  • Filed test(enrichment): add flaky-test retry to the review-enrichment suite #2425 to track a related finding: review-enrichment's test suite has no flaky-test retry (unlike the main suite's retry: 1) — observed one non-deterministic failure there during this investigation, unrelated to any change here.
  • A bigger D1-fixture win (cloning a pre-built schema file via copyFileSync instead of re-executing DDL) is possible and measured ~35x faster per call in a microbenchmark, but needs its own temp-file lifecycle/cleanup design to be safe on this repo's persistent self-hosted runner (a crash mid-suite could otherwise leak files across runs indefinitely). Left as a follow-up rather than folded into this fix.

@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.88%. Comparing base (6556a0f) to head (8a1e611).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2426   +/-   ##
=======================================
  Coverage   95.88%   95.88%           
=======================================
  Files         224      224           
  Lines       25143    25143           
  Branches     9143     9143           
=======================================
  Hits        24109    24109           
  Misses        421      421           
  Partials      613      613           
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb

loopover-orb Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-02 00:17:29 UTC

4 files · 1 AI reviewer · no blockers · readiness 86/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • Touches a guarded path — held for manual review

Review summary
This PR narrows CI path filters, caps Vitest coverage workers at 4, adds an opt-in exclusion for self-contained MCP CLI tests, and caches concatenated migration SQL for the D1 test helper. The D1 cache is behavior-preserving for the provided helper because every TestD1Database instance still executes the full migrated schema into its own in-memory database. The CI filter split is mostly coherent: backend changes still trigger OpenAPI drift checks through the new uiContract output, while UI app checks stay limited to actual UI/app/dependency changes.

Nits — 6 non-blocking
  • nit: .claude/skills/contributing-to-gittensory/SKILL.md:181 documents `npm run test:changed` against `origin/main`, but the same guide tells external contributors to add `upstream`; consider either documenting the required fetch/ref or using a less fork-sensitive base.
  • nit: test/helpers/d1.ts:17 uses a truthiness check for the migration cache, so an empty migration directory would re-read on every construction; `migratedSql !== null` is the tighter sentinel even if this repo always has migrations.
  • nit: .github/workflows/ci.yml:171 hard-codes `--maxWorkers=4` based on today’s runner quota; add a short note about where to update this when the self-hosted runner size changes.
  • In test/helpers/d1.ts:17, change the cache guard to `if (migratedSql !== null) return migratedSql;` so the sentinel matches the declared type.
  • In .claude/skills/contributing-to-gittensory/SKILL.md:181, mention that `origin/main` must exist and be current, or switch the documented command to the repo’s established upstream flow.
  • Touches a guarded path — held for manual review — A maintainer must review and merge this change.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (size label size:XS; no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 65 registered-repo PR(s), 55 merged, 572 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 65 PR(s), 572 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 65 PR(s), 572 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Triage stale or unlinked PRs.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 1, 2026
@JSONbored
JSONbored force-pushed the claude/speed-up-test-d1-fixture branch 4 times, most recently from 2eb3d59 to 71fb241 Compare July 1, 2026 23:01
JSONbored added 7 commits July 1, 2026 16:07
TestD1Database's constructor rebuilt the whole schema from scratch on every
instantiation, reading and executing all migrations/*.sql files sequentially
(~1500 call sites across the suite). Benchmarked at ~33ms/call, almost
entirely migration replay -- a meaningful, avoidable share of the full
test:coverage run.

Build the migrated schema once per worker process, serialize it, and clone
that snapshot for every subsequent instance via node:sqlite's serialize()/
deserialize() instead of re-executing every migration file each time.
Measured locally: ~1000x faster per call (33ms -> 0.03ms); full-suite
coverage run time dropped from 2:07 to 0:38 with identical coverage numbers.
…cope

Two independent CI-cost fixes found while profiling why validate-code was
taking 6-10 minutes per PR on the self-hosted runner:

- Test with coverage used --maxWorkers=100%. Node's os.cpus() reports the
  HOST's full core count inside a container, ignoring the Docker cgroup CPU
  quota, so this oversubscribed the vitest worker pool far past what the
  runner can actually execute in parallel (confirmed: `docker exec` into
  the runner reports 8 cores while its Docker CPU limit was 2, since raised
  to 4). Pin to --maxWorkers=4 to match the runner's real quota instead of
  a number Node can't correctly derive.

- The `ui` changes filter included `src/**`, so any backend-only PR ran the
  full ui:lint/typecheck/test/build toolchain even when it couldn't
  possibly affect the UI. Split it: `ui` (the UI app/extension's own code,
  plus dependency bumps) still triggers the full toolchain; a new
  `uiContract` category (src/**, the openapi-writer script) triggers only
  the lightweight ui:openapi:check drift check. If a backend change is
  genuinely UI-relevant, regenerating openapi.json per CONTRIBUTING.md
  lands a change under `ui` and correctly re-triggers the full toolchain
  anyway. UI/MCP version audit is re-scoped to `ui || mcp` (it checks an
  MCP package-version string, never the OpenAPI contract, so it was never
  really a `src/**`-driven check to begin with).
vitest run --changed=<ref> selects only tests whose module graph is
affected by the diff against that ref, via real import-graph analysis
(not just path globs). Useful for the local edit-test-fix loop; the
authoritative pre-push check stays npm run test:coverage / test:ci —
this is not wired into CI or the Codecov gate.
test/unit/mcp-cli-*.test.ts, mcp-discovery.test.ts, and mcp-output-schemas.test.ts
(~115s combined) are slow -- subprocess-per-test for the CLI-harness files, real
MCP protocol handshakes for the others -- and were unconditionally bundled into
Test with coverage on every backend PR, regardless of whether anything relevant
to them changed.

Their real dependency surface is narrower than "any backend change": the
CLI-harness tests spawn packages/gittensory-mcp against a fixture server (never
importing src/ directly, confirmed by grep) seeded via src/db/repositories and
src/registry/*, and mcp-discovery/mcp-output-schemas exercise src/mcp/server.ts
in-process. Added a new mcpCliCluster changes-filter scoped to exactly those
paths, and skip the cluster (via vitest's --exclude) on a PR where it's false.
Push events always run the full suite unchanged, matching the existing policy
for every other path-filtered step.

This is safe in a way a naive "just skip unrelated tests" isn't: skipping a
test can only ever make Codecov's patch-coverage number look WORSE, never let
a real regression through silently. If a PR's changed lines happen to be
covered only by this cluster, patch coverage just fails loudly and the fix is
to touch one of the mcpCliCluster paths (already true for any PR that
legitimately affects this code) or run the cluster manually.
…anged

Two review findings from the Gittensory Orb Review Agent on this same PR:

- Blocker: mcpCliCluster omitted package.json/package-lock.json, so a
  dependency bump could change MCP CLI/server runtime behavior (the MCP
  SDK, node:sqlite, etc.) and still skip the only test cluster that
  exercises it. Unlike a changed-lines gap, that risk is NOT caught by
  Codecov's patch-coverage number, so the earlier "skipping can only make
  coverage look worse" safety argument didn't cover this case. Added both
  files to the filter, matching how the pre-existing mcp filter already
  treats them as relevant.
- Nit: test:changed had no documented workflow. Added a mention to the
  contributing skill's "iterate fast" guidance.
Third review finding on this same filter (after the missing dependency
files fix): omitted .github/workflows/**, test/helpers/**, vitest*.config.ts,
and migrations/** -- so a change to the skip logic itself, the shared D1
test fixture (the very thing this PR's first commit touches), Vitest's
runtime config, or DB schema could all set SKIP_MCP_CLI_CLUSTER=true while
changing something this cluster actually depends on.

Patching each flagged path individually was the wrong shape of fix -- it
kept missing things because it was re-deriving a second, narrower opinion
of "what's backend-relevant" instead of reusing the one `backend` already
has. Redesigned: mcpCliCluster is now backend's filter verbatim for every
category except src/**/test/**, which are narrowed to the specific files
this cluster touches (that's where the actual savings are: most backend
PRs touch some src/test file, but rarely these specific ones). Every other
category -- config, scripts, deps, migrations, the workflow file itself --
is copied unchanged, so it can't drift out of sync with what `backend`
already knows can affect a test run.
…iant

Non-blocking nits from the same review pass:
- Cross-reference comments between backend and mcpCliCluster so a future
  edit to one is more likely to prompt mirroring the other.
- Document that the migrated-schema snapshot cache assumes migrations/*.sql
  is read-only for the life of a worker process; nothing currently mutates
  a migration file in-process, but this makes the assumption explicit.
@JSONbored
JSONbored force-pushed the claude/speed-up-test-d1-fixture branch from 71fb241 to 6179ec3 Compare July 1, 2026 23:10
JSONbored added 2 commits July 1, 2026 17:09
DatabaseSync.prototype.serialize()/deserialize() do not exist on Node 22 (this
repo's .nvmrc) at all -- confirmed absent on 22.23.1, present only from Node
24+. The snapshot-clone cache added in the previous commit passed locally on a
newer Node but crashed every test in CI (`db.deserialize is not a function`),
since CI runs the pinned Node 22 on the self-hosted runner.

Cache the concatenated migration SQL instead of a serialized snapshot: still
avoids re-reading and re-sorting ~90 migration files on every one of the
~1500 TestD1Database construction sites, using only stable, always-available
DatabaseSync/fs APIs. Verified green on both Node 22.23.1 (matching CI) and
the local Node, with identical coverage numbers.
…endency

mcpCliCluster's hand-picked src/** allowlist (src/mcp/**, src/auth/security.ts,
src/db/repositories.ts, src/registry/normalize.ts, src/registry/sync.ts,
src/services/repo-outcome-patterns.ts) was meant to gate when the whole
cluster could be safely skipped. It missed the mark for
mcp-output-schemas.test.ts specifically: that file imports src/mcp/server.ts
in-process, and server.ts alone directly imports ~40 other src/ modules
(src/github/app.ts, src/signals/slop.ts, src/settings/autonomy.ts,
src/orb/analytics.ts, most of src/services/** and src/signals/**) that were
never in the allowlist. A backend change to any of those files would skip
mcp-output-schemas.test.ts with no rerun to catch a regression, while
`backend` itself still treated the same change as test-relevant -- a reachable
CI blind spot on a required, one-shot gate.

Verified by direct-import inspection that the other 6 files in the cluster
(test/unit/mcp-cli-*.test.ts and mcp-discovery.test.ts) are genuinely
self-contained: they import nothing from root src/, only spawning the built
packages/gittensory-mcp CLI as a subprocess. Split the filter accordingly:
mcpCliHarness now scopes only those 6 self-contained files, mirroring the
existing `mcp` filter's own self-contained-package trust boundary.
mcp-output-schemas.test.ts is no longer excludable at all -- it always runs
whenever `backend` does, same as the rest of the suite.
@JSONbored
JSONbored merged commit 9d58403 into main Jul 2, 2026
12 checks passed
@JSONbored
JSONbored deleted the claude/speed-up-test-d1-fixture branch July 2, 2026 00:21
@github-project-automation github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant