[BUG] Session history lost on server restart — Fixed with OpenClaw assistance

[Triyarms]

Diagnosed, fixed, and verified by OpenClaw AI agent.

1. Problem Description

When the Hermes Web UI server is restarted (via terminal or crash) during an active agent task, the session history is lost. After restart:

• The UI shows "Connection Error" or an empty chat.
• Only the last user message is visible.
• The assistant's partial response and tool usage history are lost.

2. Root Cause Analysis

The issue is a combination of server-side and client-side problems:

1. Server-side: During streaming, messages are held in memory. If the server restarts, these are lost unless explicitly saved to disk.
2. Server-side: active_stream_id persists in session JSON files, causing the client to attempt reconnecting to a dead stream.
3. Client-side: loadSession in sessions.js would clear S.messages on connection errors.
4. Client-side: boot.js lacked retry logic for server restart scenarios.

3. The Comprehensive Fix

We applied fixes across 5 files to ensure full session persistence and proper reconnection:

A. api/models.py — Read from Disk, not Cache
Bypasses the in-memory cache (SESSIONS) when loading full session data.

--- a/api/models.py
+++ b/api/models.py
@@ -674,6 +674,15 @@ def get_session(sid, metadata_only=False):
      messages list. Use this when you only need compact() metadata and not the
      actual message history (e.g., for fast sidebar switching).
      """
+    # If loading full session (with messages), skip in-memory cache to avoid
+    # returning stale data that hasn't been flushed to disk yet.
+    if not metadata_only:
+        s = Session.load(sid)
+        if s:
+            with LOCK:
+                SESSIONS[sid] = s
+                SESSIONS.move_to_end(sid)
+            return s
     with LOCK:
         if sid in SESSIONS:
             SESSIONS.move_to_end(sid)  # LRU: mark as recently used

B. api/streaming.py — Periodic Save During Streaming
Saves the session every 30 seconds during streaming.

--- a/api/streaming.py
+++ b/api/streaming.py
@@ -1391,6 +1391,24 @@ def _run_agent_streaming(
     # Register this stream with the global streaming meter
     meter().begin_session(stream_id)
 
+    # ── Periodic Session Save (prevents loss on restart) ──
+    _save_stop = threading.Event()
+    _partial_save_interval = 30  # seconds
+
+    def _periodic_save():
+        while not _save_stop.wait(_partial_save_interval):
+            try:
+                if s and s.messages:
+                    s.save()
+                    logger.debug('Periodic session save completed')
+            except Exception as e:
+                logger.debug(f'Periodic save failed: {e}')
+
+    _save_thread = threading.Thread(target=_periodic_save, daemon=True)
+    _save_thread.start()
+
     # Metering ticker... (rest of the code)

C. server.py — Clear Stale Stream IDs on Startup
Clears active_stream_id from all session files on server startup.

--- a/server.py
+++ b/server.py
@@ -XX,6 +XX,18 @@ def main():
     STATE_DIR.mkdir(parents=True, exist_ok=True)
     SESSION_DIR.mkdir(parents=True, exist_ok=True)
 
+    # ── Clear stale active_stream_id on startup ──
+    try:
+        import json as _json
+        _cleared = 0
+        for _f in SESSION_DIR.glob('*.json'):
+            try:
+                _d = _json.loads(_f.read_text(encoding='utf-8'))
+                if _d.get('active_stream_id'):
+                    _d['active_stream_id'] = None
+                    _f.write_text(_json.dumps(_d, ensure_ascii=False, indent=2), encoding='utf-8')
+                    _cleared += 1
+            except Exception: pass
+        if _cleared: print(f'  [startup] Cleared active_stream_id from {_cleared} session(s) after restart.', flush=True)
+    except Exception as e: print(f'  [startup] Warning: failed to clear stale stream IDs: {e}', flush=True)
+
     # Start the gateway session watcher...

D. static/sessions.js & static/boot.js
Prevents clearing messages on error and adds retry logic.

4. Verification Steps

1. Apply all patches.
2. Start the server.
3. Ask the agent a long task (e.g., "Write 5 anecdotes").
4. While the agent is responding, restart the server (kill + python3 server.py).
5. Expected Result: Server clears stale stream IDs, browser retries connection, full history is visible.

[nesquena-hermes]

Thanks for the detailed write-up — the diagnosis lines up with shapes we've seen before, and the four-file decomposition (server-side persistence, stream-id cleanup, client-side resilience) is the right direction. A few notes before this becomes a PR though, because some of the proposed mechanics need closer review:

1. api/models.py cache bypass — direction good, mechanism risky. Bypassing SESSIONS cache on every get_session(metadata_only=False) call replaces a stale-read risk with a disk-read storm. The active-pane reattach path hits get_session() on every page load and on every cross-tab sync; reading from disk every time will hurt sidebar responsiveness on profiles with hundreds of sessions. The right shape is probably invalidate-on-write (or version stamp) rather than always-read-from-disk. If the streaming path persists, the cache should be the source of truth.

2. api/streaming.py 30s periodic save — needs lock semantics. A daemon thread calling s.save() every 30s while _run_agent_streaming is also mutating s.messages will race against the writer unless save() already takes LOCK (or s is per-session pinned). Tearing-write of the JSON file under load is a worse outcome than the original loss-on-restart problem. The fix needs to use the same lock the streaming path uses, OR the save needs to be triggered from the existing message-append code path rather than a separate thread.

3. server.py startup clear of active_stream_id — works, but the placement matters. Walking SESSION_DIR.glob('*.json') on startup is fine for single-digit-thousand sessions. For larger profiles this is multi-second startup latency. Consider deferring (lazy-clear on first get_session after restart) or only clearing sessions whose active_stream_id is older than the server's start time.

4. Client-side retry in boot.js — needs to distinguish between "server temporarily unreachable" (retry) and "server returned 401" (redirect to /login, not retry). The 401 case is already handled by static/workspace.js api() wrapper; make sure the retry doesn't fight it.

5. The loadSession "don't clear on error" change — this is actually subtle. Today's clear-on-error is intentional in some paths because stale messages from a deleted/rotated session shouldn't persist in the UI. The fix needs to scope the no-clear to "transient network error" only, not "session-not-found".

Concrete suggestion: open this as a PR (one PR per file ideally, but a single one is fine if cohesive). Include a regression test that simulates the kill-during-stream scenario — tests/test_session_persistence_on_restart.py could mock the streaming path, write partial state, simulate restart, and assert that get_session() returns the partial-but-correct messages array. That's the test that pins the invariant you're trying to establish.

Also worth cross-linking with #1466 (in-flight session switching reliability) and PR #1467 (just merged the same-stream reuse invariant) — those overlap on the "what does an in-flight session look like across page lifecycle events" question, and a shared mental model helps avoid duplicate fixes.

I'll leave this open as a tracked issue. Filed under bug + needs-pr in spirit.

[ai-ag2026]

I reproduced an adjacent stale-runtime-state variant while debugging a production WebUI session:

• /health reported active_streams: 0
• the session JSON still had a dead active_stream_id
• stale pending_user_message / pending_attachments / pending_started_at also survived
• browser-side INFLIGHT[sid] could keep the UI in a busy/thinking state even after the server reported the session idle

I have a smaller surgical branch prepared locally against current origin/master that focuses on this part of the issue:

• backend helper clears stale active_stream_id + pending runtime fields when the stream is no longer in the live STREAMS registry
• cleanup runs before session metadata/status is returned and before starting a new chat turn after a stale-id conflict
• frontend drops local INFLIGHT[sid] when the loaded server session has no active_stream_id
• service-worker cache name is bumped so the frontend fix is delivered
• 5 regression tests pass

This does not attempt to solve WAL/partial-token crash persistence. It just removes dead runtime state so the UI does not reconnect to a stream that cannot exist anymore.

[ai-ag2026]

Additional focused slice for this recovery class: #1532.

It covers the case where a WebUI-origin session is present in canonical Hermes state.db but missing its WebUI JSON sidecar. The bridge previously had a hard source != 'webui' predicate, so those rows stayed invisible even when the caller opted into agent-session import. This is separate from stale runtime stream state (#1525).

[nesquena-hermes]

Thanks @ai-ag2026 — that's a clean decomposition of this umbrella into surgical, separately-reviewable slices. Two PRs landed against this issue today:

PR	Slice	What it does
#1525	Stale runtime stream state	Backend _clear_stale_stream_state() helper + frontend INFLIGHT drop + cache bump. Clears dead active_stream_id / pending_user_message / pending_attachments / pending_started_at when the stream is no longer in STREAMS. 5 regression tests.
#1532	WebUI-origin sidecar gap	Removes the hard s.source != 'webui' predicate in read_importable_agent_session_rows() so WebUI-origin sessions present in state.db but missing their JSON sidecar are recoverable through the bridge. 1 regression test.

Both are well-scoped, defensive, and don't try to solve WAL/partial-token persistence (which is the harder problem and is being tracked separately under #1353). I'll do an initial review pass on each and route them through the next contributor batch.

What this leaves on the umbrella:

1. WAL / partial-token persistence during streaming — the original OpenClaw write-up's biggest piece. Fix/chat history WAL recovery #1353 (JKJameson) covers a chunk of this; needs an independent review pass before it can land. Not blocking your two PRs.
2. Periodic save during streaming — the OpenClaw 30s-thread proposal. Still needs the lock-semantics rework I flagged in the first comment. Could be a separate PR after fix: clear stale WebUI stream state #1525 lands.
3. Server.py stale-stream sweep on startup — your fix: clear stale WebUI stream state #1525 makes this less urgent (lazy clear on first get_session-like access already happens via the helper), but the explicit-startup-clear is still nice belt-and-suspenders for big profiles. Optional follow-up.

I'll keep this issue open as the umbrella and close out individual slices as their PRs ship. Appreciate the focused, test-backed approach.