Skip to content

Add SQLite FTS5 scrollback index (no UI yet) - #29

Merged
buggerman merged 1 commit into
mainfrom
scrollback-fts5-index
Apr 25, 2026
Merged

Add SQLite FTS5 scrollback index (no UI yet)#29
buggerman merged 1 commit into
mainfrom
scrollback-fts5-index

Conversation

@buggerman

Copy link
Copy Markdown
Owner

Summary

  • First half of scrollback search: a SQLite + FTS5 index that mirrors the writes ScrollbackStore already performs to per-channel JSONL. Adds the indexer, the search/clear API, and idempotent cold-start backfill that runs through the existing rehydrate path. No UI in this PR — that lands in a follow-up so this can be reviewed and reverted independently.
  • Pure-Swift policy is honored under the carveout merged in Allow imports of system C modules (e.g. SQLite3) in AGENTS.md #28: the unsafe sqlite3_* C surface lives entirely inside an actor and the public API takes/returns Swift value types only (Message, UUID, Date, String).
  • Phrase, prefix, boolean, and column-scoped queries come for free via FTS5 ("foo bar", bar*, term1 AND term2 NOT term3, sender:alice).

Changes

  • Sources/BryggaCore/Persistence/ScrollbackIndex.swift — new file. Public actor ScrollbackIndex with index(_:serverID:target:), search(_:scope:limit:), and clear(serverID:target:). Public SearchScope enum and SearchHit struct. Schema: one messages FTS5 virtual table with sender/content indexed and msg_id/server_id/target/timestamp/kind UNINDEXED, tokenizer unicode61 remove_diacritics 2. Database opens lazily on first call; PRAGMA journal_mode=WAL + synchronous=NORMAL. Idempotent on msg_id so cold-start replay never double-counts.
  • Sources/BryggaCore/IRC/IRCSession.swift — three new index sites alongside the existing ScrollbackStore.append: record(_:in:), recordServer(_:), and the chathistory finalize loop. loadScrollbackIfNeeded(for:) now also indexes loaded messages so channels first seen post-launch (JOIN, openQuery) backfill into the index.
  • Sources/BryggaCore/Models/AppState.swiftrestoreFromStore's rehydrate task indexes every loaded message after insertion (cold-start backfill on the canonical channel name). closePrivateMessage and removeServer now clear the index for the closed scope. Backfill is "lazy through the rehydrate loop" rather than a filesystem walker, so we always use real channel names instead of the sanitized JSONL filenames.
  • Tests/ScrollbackIndexTests.swift — new file. 10 unit tests covering single-message search, idempotency, server/channel scope filtering, clear semantics, FTS5 phrase + prefix + column-scoped queries, timestamp/kind round-trip, and empty-query handling. Each test uses :memory: for full isolation.

Test plan

  • swift build — passes
  • swift test — 131 tests pass (+10 new)
  • swiftformat --lint . — clean
  • Manual:
    • Connect to a running ircd, exchange a few messages, then in an lldb session call await ScrollbackIndex.shared.search("...") to confirm the index is being populated.
    • Verify ~/Library/Application Support/Brygga/scrollback.sqlite is created and contains rows: sqlite3 scrollback.sqlite 'SELECT count(*) FROM messages'.
    • Restart the app — confirm the rehydrate path back-fills the index for previously-seen channels (idempotent: count should not double).
    • Close a PM (Cmd+W) and confirm its rows are gone from the index. Same for "Remove Server".

Risk / rollback

  • New file at ~/Library/Application Support/Brygga/scrollback.sqlite. Disk footprint scales with scrollback volume; FTS5 typically uses ~1.5–2× the indexed text size. Safe to delete the file at any time — the next launch's rehydrate will repopulate.
  • All write sites are fire-and-forget after the existing ScrollbackStore.append so a failing index write can't break the existing JSONL pipeline.
  • The unsafe pointer wrapping is contained inside the actor — OpaquePointer and sqlite3_* calls are not visible to the rest of the codebase.
  • No deinit-time cleanup (would require isolated deinit from macOS 15.4); production singleton never deinits, :memory: test instances cleaned up at process exit. Documented in-source.
  • Revert: this single commit. The scrollback.sqlite file becomes orphaned on disk after revert; users can delete it manually with no impact on chat history (which still lives in JSONL).

@buggerman
buggerman merged commit e20f33c into main Apr 25, 2026
1 check passed
@buggerman
buggerman deleted the scrollback-fts5-index branch April 25, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant