[Agent] Worker-pool wedge observability: wedged-command count on heartbeat + shorter watchdog tier for ephemeral commands

[ToddHebebrand]

Description

Two follow-ups deliberately not taken in PR #2395 (#2387 hardening), both blocked on open PRs merging:

1. Fleet-level wedged-worker visibility: PR fix(agent): bound worker-wedging blockers that retain command payloads (#2387) #2395 adds a log-only 2h in-flight watchdog in executeCommandViaPool. The count of in-flight/overdue commands should ride the heartbeat payload so wedged workers are visible fleet-wide from the dashboard — this is exactly the signal that would have shortcut the [Agent] macOS agent retains ~2.5 GB of live, compressed heap (root cause unconfirmed — evidence + leading suspect) #2387 investigation. Natural home: the agentRuntime gauges object added by PR feat(agent): runtime memory gauges on heartbeat + on-demand pprof capture (#2389) #2394 (which lands in device_metrics.custom_metrics).

2. Shorter watchdog tier for ephemeral commands: terminal_data / tunnel_data / desktop input (isEphemeralCommand, agent/internal/heartbeat/heartbeat.go:3547) should complete in milliseconds; the 2h tier is far too coarse to catch a wedged interactive path early. Add a short tier (e.g. 60s log-only) for ephemeral command types.

Blocked On

• PR feat(agent): runtime memory gauges on heartbeat + on-demand pprof capture (#2389) #2394 (agentRuntime heartbeat gauges) — for (1)
• PR fix(agent): bound worker-wedging blockers that retain command payloads (#2387) #2395 (in-flight watchdog) — for both

Affected Files

• agent/internal/heartbeat/heartbeat.go
• agent/internal/collectors/runtime_stats.go (added in feat(agent): runtime memory gauges on heartbeat + on-demand pprof capture (#2389) #2394)

Refs #2387.

[ToddHebebrand]

PR: #2408 implements both halves — commandsInFlight/commandsOverdue gauges on the heartbeat's agentRuntime object (persisted via the #2394 device_metrics.custom_metrics path, no migration) and a 60s log-only watchdog tier for isEphemeralCommand types alongside the 2h default. Review surfaced that the heartbeat-poll dispatch path bypassed the tracker; fixed so both delivery channels are counted, with tracking starting at worker pickup rather than enqueue.