WebUI memory leak causes the session to completely freeze

[LineTang]

Environment

• Hermes Agent: hermes-webui (latest, GitHub: nesquena/hermes-webui)
• OS: WSL2 Ubuntu 26.04 on Windows
• Hardware: Samsung Galaxy Book Flex (i5-1035G4, 12GB RAM, WSL2 memory limit 2GB)
• Model: mimo-v2.5-pro via xiaomi provider

Bug Description

After extended use, the WebUI server process (server.py) memory grows to 1GB+ RSS, causing all API endpoints to time out. The frontend then falls back to returning locally cached content — specifically the compression marker — for any user input, making the session appear completely broken.

Steps to Reproduce

1. Start WebUI on port 7002
2. Have a long conversation (500+ messages, 800KB session file) with context compression triggered
3. Continue using the WebUI for ~30+ minutes without restart
4. Observe that sending any message immediately returns [Your active task list was preserved across context compression] — no API call is made

Evidence

Process memory at time of failure

PID 15005 — started 11:38, observed at ~11:47 (9 minutes uptime)
  %CPU 17.1  %MEM 56.3  VIRT 3.4GB  RSS 1.07GB
  command: python server.py (port 7002)

API timeout

$ curl -s http://127.0.0.1:7002/api/health/agent
# Timed out after 60s — no response

Crash diagnostics (auto-collected by keepalive)

crash-20260514_114711/ — WebUI health check failed, keepalive restarted

Session data (backend is healthy)

• 11 independent sessions all triggered context compression at message index 104
• After compression, the backend continued normally — all sessions show valid tool calls and assistant responses for 400+ messages after compression
• The problem is purely frontend: when server.py is unresponsive, the browser-side code returns cached content without attempting an API call

Affected session

session_20260514_065326_8a3de3.json — 544 messages, 796KB, mimo-v2.5-pro

Expected Behavior

• WebUI should not leak memory during normal use
• If the API is unreachable, the frontend should show an error toast/banner, not silently return stale content
• The compression marker [Your active task list was preserved...] should never be returned as a "response" to user input

Actual Behavior

• server.py RSS grows to 1GB+ (56% of system memory) within ~9 minutes
• All API calls time out
• Frontend returns the compression marker text as if it were the model's response, with no API round-trip
• User sees the same message repeated for every input — appears as if the model is "stuck"

Workaround

Kill the WebUI process and let keepalive restart it:

kill $(pgrep -f "server.py.*7002")
# keepalive auto-restarts within 5s

Additional Notes

• The session file itself is not corrupted — all 544 messages are valid
• The compression itself works correctly — the backend continues normally after compression
• The _isPreservedCompressionTaskListMessage() function in ui.js:4544 detects this specific message pattern, which suggests the frontend has special handling for it — but when the server is unreachable, this content leaks through as a "response"
• This may be related to how the frontend handles SSE stream disconnection when the server is overloaded

[nesquena-hermes]

@LineTang thanks for the detailed report and crash-diagnostics paths — labeling this performance + needinfo and parking on Critical Bug Fix until we have a memory profile we can act on.

What we need to root-cause this

The symptoms (RSS climb to 1GB+ in ~9 min, all endpoints timing out, frontend falling back to cached compression-marker text) are consistent with at least three plausible bug classes, and they have different fixes:

1. Per-session leak in the streaming hot path — INFLIGHT map / token accumulator / SESSIONS cache retains references after a turn ends, and a long conversation with frequent compressions amplifies it.
2. Tracemalloc-style transient blowup during compression — the 11-session-compressed-at-msg-104 detail is striking. Compression itself materializes the full transcript in memory; if 11 of them ran near-simultaneously on a 2GB-cap WSL2 box, that alone could push RSS to 1GB temporarily.
3. Frontend cached-fallback misclassification — separate from the leak, the fact that the frontend returns the compression marker as if it were a model response when the API is unreachable is its own bug (we should show an error banner instead). This part we can fix independently of the memory leak root cause.

What would unblock us

If you can grab one of these next time you reproduce, that's enough to pick a direction:

Easiest — py-spy dump (no install on target):

pip install py-spy  # one-time
sudo py-spy dump --pid $(pgrep -f 'server.py') > py-spy-dump.txt

Attach py-spy-dump.txt. This tells us what every thread is doing the moment RSS hits 1GB — usually fingers the leak in a few minutes of reading.

Better — tracemalloc snapshot if you can restart with one env var:

PYTHONTRACEMALLOC=10 python server.py
# then when it hits 1GB:
curl -s http://127.0.0.1:7002/api/debug/tracemalloc > tracemalloc.json

We don't have a tracemalloc endpoint yet; if you can run with PYTHONTRACEMALLOC=10 we can write a small endpoint patch you'd cherry-pick to grab the snapshot. Tell us if that's an option and we'll send one.

Or — gc.get_objects() count if you have a Python REPL into the running process via manhole/debugpy:

import gc; from collections import Counter
print(Counter(type(o).__name__ for o in gc.get_objects()).most_common(20))

What we'll do independently while we wait

The "frontend silently returns cached compression marker on API timeout" symptom is a real bug regardless of the leak — that should be a visible error toast/banner, never a fake response. I'll file that as a separate sprint-candidate so a fix can land before the leak root cause is known. Tracking it on the Critical Bug Fix milestone alongside this one.

Ping the issue when you have a py-spy dump or can run with tracemalloc enabled. Once we have either, the next step is a sprint-candidate PR.

[nesquena-hermes]

Filed the frontend half as #2262 — it can ship independently of the memory-leak root cause.