[Agent] No pprof or runtime memory instrumentation — field memory issues are undiagnosable

[ToddHebebrand]

Description

The agent has no heap profiling and no runtime memory instrumentation of any kind:

$ grep -rn 'net/http/pprof|ReadMemStats' agent/
(no matches)

The only memory-related call in the entire agent codebase is a single debug.FreeOSMemory() in remote desktop session teardown (internal/remote/desktop/session.go:437).

Impact

Diagnosing #2387 (a macOS agent holding ~2.5 GB) required inferring the leak's shape from outside the process — comparing RSS against compressed footprint, sampling the trend over 30 minutes, and writing a standalone Go harness to characterize how the darwin scavenger reports phys_footprint — because there was no way to ask the agent what was on its heap. A single -inuse_space profile would have named the retainer immediately.

For an agent targeting 10,000+ machines, "a customer reports the agent is using 2.5 GB" is a question we currently cannot answer without shell access to the affected host, and even then only by inference.

Proposed Fix

Add a gated heap-profile capability. Suggested shape:

• net/http/pprof bound to localhost only, off by default, enabled via config or a signed command — never exposed on a listening interface reachable off-box.
• Periodic runtime.ReadMemStats sampling (HeapAlloc, HeapInuse, HeapReleased, NumGoroutine) reported on the heartbeat, so leaks are visible fleet-wide from the server without touching the device.

The goroutine count in particular would have made #2387's leading suspect (unbounded go processCommand) obvious from the dashboard.

Security note

This is a root daemon. The profiling endpoint must not be reachable off-box and must not be enabled by default. Treat the enable path as privileged.

Affected Files

• agent/internal/agentapp/main.go (startup wiring)
• agent/internal/heartbeat/heartbeat.go (memstats on heartbeat payload)

[ToddHebebrand]

PR: #2394 implements both halves of the proposed fix.

Shape shipped:

• Always-on runtime.ReadMemStats + NumGoroutine gauges on every heartbeat (agentRuntime payload field), persisted into the existing device_metrics.custom_metrics jsonb — no migration, tolerant schema so a bad value never 400s the heartbeat.
• On-demand capture_pprof privileged command (clones set_log_level): heap and/or goroutine profiles captured in-process via runtime/pprof, returned base64 in the command result (go tool pprof-ready gzip protobuf), 1 MiB raw cap per profile. No pprof HTTP listener at all — nothing reachable off-box, nothing enabled by default, per the security note.

Not included: an MCP tool (capture_agent_pprof) — touches 7+ registration sites, left as a follow-up to keep the PR bounded.