Windows + Git Bash: idle CPU still scales with pane count (~0.11 cores/pane); PowerShell panes cost ~0.01

[hernant]

Is this a reproducible bug?

• I confirm this is a reproducible bug, not a feature request, idea, question, contribution proposal, or direction check.
• I reproduced this bug on the version and environment reported below using the exact steps provided.

Current behavior

The server sits at ~3.1 of 8 logical cores with every pane idle — no output, no input, no agent activity.

Adding six idle Git Bash panes adds ~0.7 cores; closing them returns to baseline. Changing only terminal.default_shell to PowerShell and repeating the exact same steps adds almost nothing.

Marginal cost per idle pane, same machine, same running server:

Pane shell	Runs	Cores per idle pane
Git Bash	3	0.093, 0.113, 0.127
PowerShell	2	~0, 0.013

This is the signature of #1158, which was fixed in v0.7.5 and verified there at ~0.0007 cores/pane. It still reproduces on 0.8.0-preview when panes run Git Bash.

Expected behavior

Idle panes cost near zero CPU regardless of which shell they run — as they already do with PowerShell.

Reproduction

1. Windows with terminal.default_shell set to Git Bash, server running with idle panes.
2. Measure server CPU over 12 s:
   $p=Get-Process -Id <server-pid>; $c=$p.TotalProcessorTime.TotalSeconds; Start-Sleep 12; $p.Refresh(); ($p.TotalProcessorTime.TotalSeconds-$c)/12
3. herdr workspace create --label t, then herdr pane split --pane <root-pane> --direction right --no-focus ×5. Wait for the prompts to settle.
4. Re-measure: +0.6–0.8 cores.
5. herdr workspace close <ws>; CPU returns to baseline within ~0.1 cores.
6. Set default_shell to powershell.exe, run herdr server reload-config, repeat steps 3–5: the same six panes add ~0.08 cores total.

Baseline drifts 3.0–3.5 cores on a live session, so the A-B-A delta is the reliable figure; the before/after baselines agreed within 0.02–0.12 cores in every run.

Impact

An idle 27-pane session permanently occupies ~3 of 8 cores on this machine, around the clock, with no agent doing any work. It also scales with unrelated system load: spawning 120 idle processes with no new panes added a further ~0.2 cores.

Git Bash is the shell I need for this workflow, so the PowerShell workaround is not usable here.

Environment

• Herdr version: 0.8.0-preview.2026-08-04-d78e3d3b5126
• Update channel (stable or preview): preview
• Operating system: Windows 11 Pro for Workstations, build 26200, 8 logical cores
• Terminal: herdr TUI client (Windows)
• Shell, if relevant: Git Bash
• Relevant config, if any: default_shell = "C:\\Program Files\\Git\\bin\\bash.exe", shell_mode = "auto"

[Pimpmuckl]

@hernant Thanks for the tag on this. Funnily enough, I was tracing a similar Windows performance issue caused by a regression in #962. The symptoms look related, but this might be a separate issue since yours also happens with idle panes, the issue I was chasing is mostly with in-use panes.

So this is super helpful actually.

I’ll take a look unless @ogulcancelik gets to it first.

[hernant]

@Pimpmuckl since filing this I traced it with an instrumented build (per-call-site counters dumped to the server log every 10s), and I have a root cause plus a validated fix. Sharing so you don't have to re-trace.

It's not snapshot volume. Steady state shows snapshots at ~3/s (the 250ms cache from your #1158 fix working as intended) and probes at ~2.4/s. The cost is per-entry work computed from the cached snapshot:

Every detection tick (300–500ms), each Git Bash pane's foreground_process_group_id call reaches the escaped-agent scan in select_pane_foreground_job_with_runtime_inspection, which runs process_entry_identifies_agent — the full identification pipeline — over every process entry on the machine (~700 here). At 12 idle panes that's ~13,000 identify calls/sec. Second, smaller: process_runtime_marker's 1s negative TTL re-reads environment blocks that are immutable for a process's lifetime.

PowerShell/cmd panes exit before the scan via !shell_is_git_bash(shell), which is why the #1158 verification (both testers on PowerShell) looked fully fixed, and why this only bites Git Bash setups.

Fix — two caches, platform/windows.rs only, +54/−6:

1. Memoize the process_entry_identifies_agent verdict per (pid, name, cmdline), with capacity/retention eviction mirroring the existing runtime-marker cache. Steady-state identify work drops to ~5 calls/s (new processes only).
2. Make runtime-marker cache entries valid for the lifetime of (pid, creation_time) instead of the 1s negative TTL — a process's environment block can't change after spawn.

No behavior change: the escaped-agent scan and recovery still run on every tick, same inputs, same results.

Measured (my machine, patch on d78e3d3):

	before	after
isolated server, 12 idle Git Bash panes	1.271 cores	0.068 cores
live session, 27 panes (13 agents), idle	3.1–3.9 cores	0.16–0.18 cores
IO Other ops/s (live session)	~790,000	~120
marginal cost per added idle pane	0.09–0.13 cores	~0 (A-B-A 0.171→0.164→0.164)

Running the patched build as my daily server since; agent detection and status still behave normally.

Happy to open a fix: PR (single file, well within the intake budget), or paste the diff here if you'd rather take it from there.

Full disclosure: the investigation, instrumentation, and patch here were done by an AI agent (Claude) driving my machine, with me supervising and approving each step. The measurements are real runs on my setup, but review the diff with that authorship in mind.

[Pimpmuckl]

Thanks @hernant, this is a really cool investigation. The per-call-site counters make the regression path quite clear, and the negative-marker result adds a useful second piece.

I ended up with an independent implementation on current master and pushed it here for comparison:
https://github.com/Pimpmuckl/herdr/tree/issue/2459-windows-git-bash-idle-cpu

Rather than keeping identification verdicts across process snapshots, this makes the existing immutable ProcessSnapshot lazily own its agent-candidate indices. Every Git Bash pane using that snapshot shares one derivation. Marker correlation stays unchanged, and an empty candidate set now returns before building the all-process parent map. Or in plain English: The process scan between the panes is shared.

On my machine, the six-pane A-B-A measurements went from 0.252 and 0.270 incremental cores per Git Bash pane to 0.00236 and 0.00173. PowerShell measured 0.00108 after the change. Pane revisions and screen hashes stayed unchanged. Escaped-agent start, exit, and resume also kept stable PIDs without falling into a repeated probe loop.

The tradeoff as I see it:

• The fix I was chasing is doing is smaller and has less cache state, but repeats some work whenever the shared process snapshot refreshes.
• Your two caches remove the work across snapshots too, including the negative environment reads. Your absolute idle results are excellent. The cost is two longer-lived cache policies and more process-lifetime state to maintain.

Both approaches address the same root cause at different cache lifetimes. Your measurements suggest the longer-lived caches may still win on absolute idle CPU, I was mostly trying to keep logical complexity in check. I might have time later today to test both implementations, but can't promise.

At this point I think the final boundary is a maintainer choice. I am leaving this as a branch only, Can can then potentially pick and choose and/or come up with a better fix.

Either way, thank you for participating, this was an excellent report and any type of Windows performance improvements are just awesome to see. Hope this can land eventually on master!

[ogulcancelik]

@Pimpmuckl this approach looks good. could you send a pr? could you also verify it doesn’t regress detection for agents installed and launched through git bash?

[Pimpmuckl]

@Pimpmuckl this approach looks good. could you send a pr? could you also verify it doesn’t regress detection for agents installed and launched through git bash?

Sure I'll give it a go, just note that my testing is quite limited to Windows but I'll check codex/pi/cc via git bash

Edit: pi, codex, cc all work and work very fast inside git bash on windows with this fix applied. Opened #2494 and put the benchmarks there.

@hernant perhaps you could test it briefly if it looks good on your side of things? That'd be super cool.

Depending on the number on your specific rig, that might indicate if a more complex solution like the one you proposed with the dual cache should be something to look at.

[kangal-bot]

Implemented on master and queued for the next release. This is not available in a published Herdr release yet.