fix: handle lone surrogates in MCP write tools

[YuanYiZheXue]

Problem

MCP write tools (mempalace_add_drawer, mempalace_diary_write) fail with UnicodeEncodeError when processing strings containing lone surrogates injected by MCP clients like WorkBuddy.

Solution

Add _clean() function that removes lone surrogates using surrogatepass/surrogateignore encoding before writing to ChromaDB.

Changes

• tool_add_drawer: Clean content, wing, room, added_by before ChromaDB upsert
• tool_diary_write: Clean entry, topic, agent_name, wing before ChromaDB add
• SHA256 hash: Use surrogatepass encoding to avoid errors

Testing

mcp__mempalace__mempalace_add_drawer --wing test --room unicode --content "????"
mcp__mempalace__mempalace_diary_write --agent_name test --entry "SESSION:2026-04-27"

See docs/fix-lone-surrogate.md for full details.

[igorls]

Thanks for the surrogate fix — \udc95-style lone surrogates from MCP clients are a real problem and the _clean() approach is the right shape. Unfortunately I can't merge this PR as-is; flagging the blockers so we can land the fix properly in a fresh PR.

Destructive rebase against an old base

The diff (+181 / -213 across 2 files) is much larger than the description suggests. Most of the deletions remove recent safety fixes that landed on develop after this branch forked:

• The entire _vector_disabled system (HNSW max_elements frozen at 16K while collection grows to 100K+ entries — MCP server segfaults on every tool call #1222 SIGSEGV mitigation): _refresh_vector_disabled_flag(), _tool_status_via_sqlite(), the BM25 fallback in tool_check_duplicate, and the vector_disabled reporting in tool_search / tool_reconnect
• _HNSW_BLOAT_GUARD import and application
• _pin_hnsw_threads() calls (HNSW race fix SIGSEGV in HNSW parallel inserts — missing hnsw:num_threads on collection creation #974 / Critical Data write loop, HNSW compaction bug #965)

These were intentional fixes for known crash modes; merging this PR would re-open all of them.

Hardcoded contributor paths

Two changes can't apply to any other user:

• mcp_server.py adds logging.FileHandler(r\"C:\Users\SJC\mempalace_mcp.log\", encoding=\"utf-8\") — would crash on every non-SJC machine on Windows, and on Linux/macOS entirely
• docs/fix-lone-surrogate.md references D:\ProgramData\Python312\Lib\site-packages\mempalace\mcp_server.py and D:\gitfork\mcp_server.py — local-machine paths that don't belong in repo docs

Path forward

The actual fix is small (~10 lines: _clean() helper + use it in tool_add_drawer and tool_diary_write + surrogatepass on the SHA256 hashes). Would you be willing to open a fresh PR against current develop with just those changes? Happy to review quickly.

Removing from the v3.3.5 milestone for now.

[YuanYiZheXue]

Hi Igor, Thanks for the detailed review! I've opened a new clean PR as you suggested: #1422 Changes: - Branched off latest develop (no destructive rebase) - Only mcp_server.py is touched (+21 / -2) - No hardcoded paths - _clean() applied in tool_add_drawer and tool_diary_write before SHA256 hash and ChromaDB write Please take a look when you have a moment. Appreciate the guidance! Best, YuanYi

…

---------- Original ---------- From: ***@***.*** Subject: Re: [MemPalace/mempalace] fix: handle lone surrogates in MCP write tools (PR #1235) igorls left a comment (MemPalace/mempalace#1235) Thanks for the surrogate fix — \udc95-style lone surrogates from MCP clients are a real problem and the _clean() approach is the right shape. Unfortunately I can't merge this PR as-is; flagging the blockers so we can land the fix properly in a fresh PR. Destructive rebase against an old base The diff (+181 / -213 across 2 files) is much larger than the description suggests. Most of the deletions remove recent safety fixes that landed on develop after this branch forked: The entire _vector_disabled system (#1222 SIGSEGV mitigation): _refresh_vector_disabled_flag(), _tool_status_via_sqlite(), the BM25 fallback in tool_check_duplicate, and the vector_disabled reporting in tool_search / tool_reconnect _HNSW_BLOAT_GUARD import and application _pin_hnsw_threads() calls (HNSW race fix #974 / #965) These were intentional fixes for known crash modes; merging this PR would re-open all of them. Hardcoded contributor paths Two changes can't apply to any other user: mcp_server.py adds logging.FileHandler(r\"C:\Users\SJC\mempalace_mcp.log\", encoding=\"utf-8\") — would crash on every non-SJC machine on Windows, and on Linux/macOS entirely docs/fix-lone-surrogate.md references D:\ProgramData\Python312\Lib\site-packages\mempalace\mcp_server.py and D:\gitfork\mcp_server.py — local-machine paths that don't belong in repo docs Path forward The actual fix is small (~10 lines: _clean() helper + use it in tool_add_drawer and tool_diary_write + surrogatepass on the SHA256 hashes). Would you be willing to open a fresh PR against current develop with just those changes? Happy to review quickly. Removing from the v3.3.5 milestone for now. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: ***@***.***>