Skip to content

fix(agent): capture_pprof reports worker-wedge gauges + 30s capture throttle (#2422) - #2453

Merged
ToddHebebrand merged 2 commits into
mainfrom
fix/2422-capture-pprof-gauges-throttle
Jul 13, 2026
Merged

fix(agent): capture_pprof reports worker-wedge gauges + 30s capture throttle (#2422)#2453
ToddHebebrand merged 2 commits into
mainfrom
fix/2422-capture-pprof-gauges-throttle

Conversation

@ToddHebebrand

Copy link
Copy Markdown
Collaborator

Summary

Fixes both gaps in the capture_pprof diagnostic command (#2394/#2408 seam):

  1. Wedge gauges were always 0/0. handleCapturePprof discarded its *Heartbeat receiver and embedded collectors.CollectRuntimeStats() directly, which never populates commandsInFlight/commandsOverdue. The snapshot now goes through h.collectAgentRuntime(now) — the wrapper that wires in inFlightCommandStats — so a capture taken while chasing an overdue-commands heartbeat trend reflects the real pool state.

  2. No throttle on GC-forcing captures. Heap/all captures unconditionally ran runtime.GC(); as a server-queued command (10 concurrent / 100 queued), a burst could force back-to-back stop-the-world GCs. Added a 30s minimum interval between admitted captures using the same atomic CAS slot pattern as the heartbeat watchdog dump throttle (fix(agent): stop heartbeat watchdog from firing every heartbeat on slow links (#2386) #2392). Throttled captures fail with an explicit rate-limited error. Payload validation runs before the throttle so malformed requests don't consume the slot.

All changes are confined to handlers_diag.go / handlers_diag_test.goheartbeat.go is untouched.

Tests

  • New: TestHandleCapturePprofIncludesWedgeGauges (tracked overdue command shows up as 1/1 in the snapshot), TestHandleCapturePprofThrottled (second capture rejected, slot frees after interval), TestHandleCapturePprofValidationDoesNotConsumeSlot.
  • Existing diag tests updated to use a zero-value *Heartbeat + per-test throttle reset.
  • cd agent && go test -race ./internal/heartbeat/... — ok (41s); go build ./... clean; go vet clean.

Closes #2422

🤖 Generated with Claude Code

…es GC-forcing captures (#2422)

Two gaps in the capture_pprof diagnostic command:

1. The embedded runtime snapshot called collectors.CollectRuntimeStats()
   directly, which never populates the worker-pool wedge gauges — every
   capture reported commandsInFlight/commandsOverdue as 0/0, falsely
   showing a healthy pool to an operator chasing an overdue-commands
   heartbeat trend. Route through h.collectAgentRuntime(now) instead.

2. Every heap/all capture forces a stop-the-world runtime.GC() with no
   rate limit; as a server-queued command (10 concurrent / 100 queued),
   queued captures could degenerate into back-to-back GC pauses. Add a
   30s minimum interval between admitted captures using the same atomic
   CAS throttle pattern as the heartbeat watchdog dump (#2392), rejecting
   throttled captures with an explicit error. Payload validation happens
   before the throttle so malformed requests don't burn the slot.

Closes #2422

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 13, 2026

Copy link
Copy Markdown

Deploying breeze with  Cloudflare Pages  Cloudflare Pages

Latest commit: c429baf
Status: ✅  Deploy successful!
Preview URL: https://100990ce.breeze-9te.pages.dev
Branch Preview URL: https://fix-2422-capture-pprof-gauge.breeze-9te.pages.dev

View logs

…ry; precise rate-limit message

Review follow-ups on #2453:
- capturePprofTryAcquire is a copy of the watchdog CAS loop, not shared
  code, so the watchdog's concurrency/boundary tests don't cover it. Add
  TestCapturePprofTryAcquireConcurrent (32 racers, exactly one winner)
  and TestCapturePprofTryAcquireInterval (1ns-before rejected, exact
  interval admitted).
- Rate-limit error now says "heap captures force a stop-the-world GC" —
  goroutine-only captures are throttled too but never call runtime.GC().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ToddHebebrand

Copy link
Copy Markdown
Collaborator Author

Review run: /pr-review-toolkit:review-pr (code-reviewer, pr-test-analyzer, silent-failure-hunter)

Findings: 0 blocking across all three agents. 4 non-blocking items raised → 3 addressed in c429baf:

  • Concurrency test for capturePprofTryAcquire (the CAS loop is a copy of the watchdog's, so the watchdog's concurrency test didn't cover it) — added TestCapturePprofTryAcquireConcurrent (32 racers, exactly one winner).
  • Interval boundary pin — added TestCapturePprofTryAcquireInterval (1ns-before rejected, exact interval admitted).
  • Rate-limit error wording claimed every capture forces a GC; goroutine-only captures are throttled but never call runtime.GC() — message now says "heap captures".

Declined (with rationale):

Tests: cd agent && go test -race ./internal/heartbeat/... — ok (11 CapturePprof tests, full package green); go vet + go build ./... + gofmt clean. CI Test Agent passed on the first commit; re-running on c429baf.

Status: review-clean, awaiting maintainer merge.

@ToddHebebrand
ToddHebebrand merged commit 968d14c into main Jul 13, 2026
41 checks passed
@ToddHebebrand
ToddHebebrand deleted the fix/2422-capture-pprof-gauges-throttle branch July 13, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Agent] capture_pprof reports zero worker-wedge gauges and lacks a capture throttle

1 participant