Description
On an idle macOS device, the agent spawned 81,683 child processes over ~2 days — roughly one every 2 seconds, continuously. The processes are correctly reaped (no zombies), so this is not a fd/process leak — it is sustained, largely wasted work on every Mac in the fleet.
Evidence
The agent's stderr log was 17 MB, and 170,078 of its 170,464 lines were fork noise from these children:
breeze-agent(60647) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
(81,683 distinct child PIDs.) Sampling live children of the daemon shows the collector command fleet: log show x2, pmset -g log, networksetup -listallhardwareports, ipconfig getpacket, dscl, pwpolicy, spctl, socketfilterfw, defaults read, system_profiler.
The expensive one
agent/internal/collectors/eventlogs_darwin.go:121 runs log show --predicate ... --style json --last Nm every 5 minutes, via two parallel sub-collectors (security + hardware predicates), and again from ReliabilityCollector on its own EventLogCollector.
Timed on the affected host:
$ time log show --predicate '<hardware predicate>' --style json --last 5m
real 3.77s user 1.46s sys 0.51s
output: 2 bytes
3.8 seconds of CPU to produce 2 bytes. The source-side messageType >= error filter is working correctly (that part is fine and should stay) — but querying the macOS unified log is expensive regardless of how little it returns. We pay that cost every 5 minutes, several times over, on every Mac, to almost always get [].
Secondary: post-hoc output cap
agent/internal/collectors/command_limits.go:43-50:
cmd := exec.CommandContext(ctx, name, args...)
output, err := cmd.Output() // buffers EVERYTHING first
if len(output) > collectorCommandOutputLimit { // 4 MiB — checked AFTER the fact
return nil, fmt.Errorf("%s output too large", name)
}The 4 MiB guard runs after the full output is already in memory, and cmd.Output() grows its buffer by doubling — so a 100 MB output costs ~200 MB peak before being rejected. runCollectorLimitedOutput (used by pmset just below) does this correctly with an io.LimitReader. The unified-log path should use the same.
Proposed Fix
- Back off the
log show cadence, and/or collapse the two parallel predicates into one query.
- Use
runCollectorLimitedOutput (streaming + io.LimitReader) for log show rather than runCollectorOutput, so the cap is enforced before the bytes are buffered.
- Consider whether the reliability collector needs its own independent
EventLogCollector pass.
Affected Files
agent/internal/collectors/eventlogs_darwin.go (primary)agent/internal/collectors/command_limits.go (runCollectorOutput post-hoc cap)agent/internal/collectors/reliability_darwin.go
Found during the macOS agent memory audit (#2387).
Description
On an idle macOS device, the agent spawned 81,683 child processes over ~2 days — roughly one every 2 seconds, continuously. The processes are correctly reaped (no zombies), so this is not a fd/process leak — it is sustained, largely wasted work on every Mac in the fleet.
Evidence
The agent's stderr log was 17 MB, and 170,078 of its 170,464 lines were fork noise from these children:
(81,683 distinct child PIDs.) Sampling live children of the daemon shows the collector command fleet:
log showx2,pmset -g log,networksetup -listallhardwareports,ipconfig getpacket,dscl,pwpolicy,spctl,socketfilterfw,defaults read,system_profiler.The expensive one
agent/internal/collectors/eventlogs_darwin.go:121runslog show --predicate ... --style json --last Nmevery 5 minutes, via two parallel sub-collectors (security + hardware predicates), and again fromReliabilityCollectoron its ownEventLogCollector.Timed on the affected host:
3.8 seconds of CPU to produce 2 bytes. The source-side
messageType >= errorfilter is working correctly (that part is fine and should stay) — but querying the macOS unified log is expensive regardless of how little it returns. We pay that cost every 5 minutes, several times over, on every Mac, to almost always get[].Secondary: post-hoc output cap
agent/internal/collectors/command_limits.go:43-50:The 4 MiB guard runs after the full output is already in memory, and
cmd.Output()grows its buffer by doubling — so a 100 MB output costs ~200 MB peak before being rejected.runCollectorLimitedOutput(used bypmsetjust below) does this correctly with anio.LimitReader. The unified-log path should use the same.Proposed Fix
log showcadence, and/or collapse the two parallel predicates into one query.runCollectorLimitedOutput(streaming +io.LimitReader) forlog showrather thanrunCollectorOutput, so the cap is enforced before the bytes are buffered.EventLogCollectorpass.Affected Files
agent/internal/collectors/eventlogs_darwin.go(primary)agent/internal/collectors/command_limits.go(runCollectorOutputpost-hoc cap)agent/internal/collectors/reliability_darwin.goFound during the macOS agent memory audit (#2387).