diff --git a/docs/assets/search-box.png b/docs/assets/search-box.png new file mode 100644 index 000000000..0a85fde32 Binary files /dev/null and b/docs/assets/search-box.png differ diff --git a/docs/assets/search-results.png b/docs/assets/search-results.png new file mode 100644 index 000000000..24aa00e09 Binary files /dev/null and b/docs/assets/search-results.png differ diff --git a/docs/plans/2026-08-14-message-content-search-implementation.md b/docs/plans/2026-08-14-message-content-search-implementation.md new file mode 100644 index 000000000..1bc65a603 --- /dev/null +++ b/docs/plans/2026-08-14-message-content-search-implementation.md @@ -0,0 +1,238 @@ +# 会话消息内容搜索实施计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 让 Ctrl+K 的会话搜索在标题之外还能命中用户与助手正文，同时保持 45ms 内容查询预算、低存储和后台首建索引。 + +**Architecture:** 新增 SQLite 文档表和两个可选的 FTS5 倒排表；解析器输出经过规范化后由后台索引器增量写入；查询服务合并标题结果与正文结果并生成摘要片段；前端改为服务端过滤并显示正文摘要。 + +**Tech Stack:** Rust + SeaORM/SQLite FTS5 + Tokio；Next.js 16 + React 19 + cmdk。 + +## Global Constraints + +- 仅索引 `MessageTurn` 中 `role` 为 `User` 或 `Assistant` 的 `ContentBlock::Text`。 +- 每个文本块 UTF-8 上限 8,192 字节，截断必须在合法字符边界。 +- 搜索范围沿用现有语义：选中文件夹时限定文件夹，否则全部可见文件夹，保留 Agent 过滤。 +- 扫描模式延迟预算 45ms；可索引文本默认阈值 40MB，短词索引与 trigram 同阈值。 +- 候选召回上限 `max(1, 当前查询范围的可见会话数)`，默认不静默截断。 +- 排序：标题命中优先，正文按相关度，同相关度按 `updated_at DESC`。 +- 搜索路径不得读取转录文件，索引写入不得在搜索请求路径执行。 +- 不改动既有 `list_all_conversations` 和“文件”页签行为。 +- 遵循现有 Rust/TypeScript 代码风格；Rust 错误类型使用 `thiserror`/`DbError`。 + +--- + +## File Structure + +- Create `src-tauri/src/search/mod.rs`：公开 normalizer、query、indexer 模块。 +- Create `src-tauri/src/search/normalizer.rs`：纯文本提取、截断、哈希、短词分词。 +- Create `src-tauri/src/search/query.rs`：查询词拆分、LIKE/FTS 表达式生成。 +- Create `src-tauri/src/search/indexer.rs`：后台队列、首建、漂移和进度。 +- Create `src-tauri/src/db/entities/message_search_document.rs`。 +- Create `src-tauri/src/db/entities/search_index_state.rs`。 +- Create `src-tauri/src/db/service/message_search_service.rs`。 +- Create `src-tauri/src/db/migration/m20260814_000001_message_search.rs`。 +- Create `src-tauri/src/commands/search.rs`。 +- Create `src-tauri/src/web/handlers/search.rs`。 +- Modify `src-tauri/src/models/conversation.rs` 或 `models/mod.rs`：搜索模型。 +- Modify `src-tauri/src/lib.rs`、`db/mod.rs`、`db/entities/mod.rs`、`db/service/mod.rs`、`db/migration/mod.rs`、`web/router.rs`。 +- Modify `src/lib/api.ts`、`src/lib/tauri.ts`、`src/lib/types.ts`。 +- Modify `src/components/conversations/search-command-dialog.tsx`。 +- Modify `src/contexts/search-dialog-context.tsx` 附近的前端状态或新建 hook。 +- Modify `i18n/messages/*.json`。 + +## Task 1: 数据模型与迁移 + +**Files:** +- Create: `src-tauri/src/db/entities/message_search_document.rs` +- Create: `src-tauri/src/db/entities/search_index_state.rs` +- Create: `src-tauri/src/db/migration/m20260814_000001_message_search.rs` +- Modify: `src-tauri/src/db/entities/mod.rs` +- Modify: `src-tauri/src/db/migration/mod.rs` +- Modify: `src-tauri/src/db/entities/prelude.rs` + +**Interfaces:** +- Produces entities `MessageSearchDocument` 和 `SearchIndexState`，表名 `message_search_document`、`search_index_state`。 +- Produces migration name `m20260814_000001_message_search`，放在 `m20260807_000001_work_task_scheduled_at` 之后。 +- `message_search_document` 列：`id`、`conversation_id`、`text`、`content_hash`、`source_ended_at`、`source_message_count`、`updated_at`；唯一索引 `idx_message_search_document_conversation`。 +- `search_index_state` 单行表，主键 `id` 带 CHECK 约束，列 `schema_version`、`mode`、`threshold_mb`、`short_fts_enabled`、`short_threshold_mb`、`scan_ms_per_mb`、`indexed_conversation_count`、`last_calibration_at`、`last_backfill_at`、`user_enabled`（默认 1）、`user_mode`（默认 `auto`）。 +- 迁移用 raw SQL 创建两个空 FTS5 表： + +```sql +CREATE VIRTUAL TABLE message_search_trigram USING fts5( + text, content='', contentless_delete=1, detail=none, tokenize='trigram' +); +CREATE VIRTUAL TABLE message_search_short USING fts5( + words, bigrams, content='', contentless_delete=1, detail=none, + tokenize='unicode61 remove_diacritics 2' +); +``` + +- [ ] 参考 `m20260803_000001_token_usage.rs` 和 `conversation.rs` 定义两个实体，时间列使用 `DateTimeUtc`。 +- [ ] 参考 `m20260803_000001_token_usage.rs` 写 `up`，用 `Table::create()` 建两张普通表；再用 `manager.get_connection().execute_unprepared(...)` 建两个虚拟表和 `search_index_state` 初始行。 +- [ ] 写 `down` 删除两张虚拟表和两张普通表。 +- [ ] 在 `entities/mod.rs`、`service/mod.rs` 暂时只注册实体模块，在 `migration/mod.rs` 注册迁移。 +- [ ] 运行 `cargo check --features test-utils`，再运行 `cargo test --features test-utils db::` 确认迁移可执行。 +- [ ] 提交：`feat(search): add message search schema and migration`。 + +## Task 2: 文本规范化与查询表达式 + +**Files:** +- Create: `src-tauri/src/search/mod.rs` +- Create: `src-tauri/src/search/normalizer.rs` +- Create: `src-tauri/src/search/query.rs` +- Modify: `src-tauri/src/lib.rs`（注册 `pub mod search;`） + +**Interfaces:** +- `normalize_turns(turns: &[MessageTurn]) -> NormalizedDocument`，`NormalizedDocument { text: String, content_hash: String }`。 +- `short_index_tokens(text: &str) -> ShortIndexTokens`，`ShortIndexTokens { words: String, bigrams: String }`。 +- `split_terms(query: &str) -> Vec`，最多 8 个非空词。 +- `escape_like(term: &str) -> String`，转义 `\`、`%`、`_`。 +- `like_pattern(term: &str) -> String`，返回 `%escaped%`。 +- `trigram_expression(term: &str) -> Option`，长度少于 3 个 Unicode 字符时返回 `None`；否则生成相邻三字符带引号短语，引号按 FTS5 规则加倍，并用 ` AND ` 连接。 +- `short_query(term: &str) -> ShortTermQuery`，枚举 `CjkUnigram`、`CjkBigram { phrase: String }`、`LatinPrefix { token: String }`。 + +- [ ] 先写 `src-tauri/src/search/normalizer.rs` 的 `#[cfg(test)]` 测试：用户/助手文本保留、Thinking/System/Tool 丢弃、空块丢弃、8192 截断、SHA-256 稳定、中文单字/二字词和拉丁词分词。 +- [ ] 运行 `cargo test --features test-utils search::normalizer`，确认先失败。 +- [ ] 实现 normalizer；CJK 判定覆盖 U+3400-U+9FFF、U+F900-U+FAFF、U+20000-U+2FA1F、平假名/片假名、谚文。拉丁分词用 `char::is_alphanumeric` 手写扫描，避免额外依赖。 +- [ ] 运行测试确认通过。 +- [ ] 同样以 TDD 实现 `query.rs` 的转义、词拆分和三字短语生成，测试包含中文、引号、`%`、`_`、少于 3 字符的词。 +- [ ] 运行 `cargo fmt`、`cargo clippy --all-targets --features test-utils -- -D warnings`。 +- [ ] 提交：`feat(search): add transcript normalization and query builders`。 + +## Task 3: 文档与状态 Service + +**Files:** +- Create: `src-tauri/src/db/service/message_search_service.rs` +- Modify: `src-tauri/src/db/service/mod.rs` +- Modify: `src-tauri/src/db/error.rs`（如需） + +**Interfaces:** +- `ensure_search_state(conn) -> Result`。 +- `get_search_state(conn) -> Result`。 +- `set_search_mode(conn, mode: &str, short_fts_enabled: bool) -> Result<(), DbError>`。 +- `upsert_document(conn, conversation_id: i32, doc: &NormalizedDocument, source_ended_at: Option, source_message_count: i32, sync_trigram: bool, sync_short: bool) -> Result`，返回文档 `id`。 +- `delete_document(conn, conversation_id: i32, sync_trigram: bool, sync_short: bool) -> Result<(), DbError>`。 +- `total_indexed_text_bytes(conn) -> Result`。 +- `visible_conversation_count(conn, folder_ids: Option>, agent_type: Option) -> Result`。 +- `list_documents_by_conversation(conn, ids: &[i32]) -> Result, DbError>`。 + +- [ ] 写失败测试覆盖：插入/更新保留 id、删除同步 FTS rowid、无孤儿行、重复会话唯一索引、单行 state。 +- [ ] 用 `conn.transaction()` 保证文档与 FTS 写原子；contentless 表删除用 `DELETE FROM ... WHERE rowid = ?`，插入用 `INSERT INTO ...(rowid, ...) VALUES(?, ...)`。 +- [ ] 运行 `cargo test --features test-utils db::service::message_search_service`。 +- [ ] 提交：`feat(search): add document and index-state services`。 + +## Task 4: 查询服务、模型与双模式命令 + +**Files:** +- Modify: `src-tauri/src/models/conversation.rs`（或新增 `models/search.rs`） +- Modify: `src-tauri/src/models/mod.rs` +- Create: `src-tauri/src/commands/search.rs` +- Create: `src-tauri/src/web/handlers/search.rs` +- Modify: `src-tauri/src/lib.rs`（invoke handler） +- Modify: `src-tauri/src/web/router.rs` + +**Interfaces:** +- 模型 `SearchMatchKind`、`DbConversationSearchResult`，字段与规格一致。 +- `search_conversations_core(conn, folder_ids: Option>, agent_type: Option, query: String, limit: u64) -> Result, AppCommandError>`。 +- `get_search_index_status_core(conn) -> Result`。 +- `#[tauri::command] search_conversations(...)`、`get_search_index_status(...)`；HTTP 参数结构 `SearchConversationsParams`。 + +- [ ] 查询服务先实现扫描模式：每词 `SELECT conversation_id FROM message_search_document WHERE text LIKE ? ESCAPE '\'`，Rust 求交集；标题结果复用 `conversation_service::list_all`。 +- [ ] 实现 FTS 模式：状态 `mode == "fts"` 时 3 字符以上词使用 `message_search_trigram`，按 `bm25` 取候选上限 `max(1, visible_conversation_count)`；1 到 2 字符词在 `short_fts_enabled` 时使用 `message_search_short`，否则扫描文档表。 +- [ ] join `conversation` 复用可见性过滤；标题优先合并；生成 snippet 时只取最终 50 条文本，用 Unicode lowercase 副本定位，返回 `snippet_prefix/match/suffix`。 +- [ ] 写测试覆盖标题与正文合并、文件夹/Agent 过滤、多词交集、LIKE 转义、FTS 三字候选、空查询、50 条上限和 snippet 边界。 +- [ ] 运行 `cargo test --features test-utils commands::search`、`cargo check --no-default-features --bin codeg-server`。 +- [ ] 提交：`feat(search): add content search query service and endpoints`。 + +## Task 5: 后台索引器与事件钩子 + +**Files:** +- Create: `src-tauri/src/search/indexer.rs` +- Modify: `src-tauri/src/search/mod.rs` +- Modify: `src-tauri/src/app_state.rs` +- Modify: `src-tauri/src/bin/codeg_server.rs` +- Modify: `src-tauri/src/lib.rs` +- Modify: `src-tauri/src/commands/conversations.rs` +- Modify: `src-tauri/src/db/service/import_service.rs`（如需） +- Modify: `src-tauri/src/acp/lifecycle.rs` 或 manager 的 TurnComplete 路径（仅投递钩子） +- Modify: `src-tauri/src/web/event_bridge.rs`（新增 `SEARCH_INDEX_PROGRESS_EVENT`） + +**Interfaces:** +- `MessageSearchIndexer::spawn(conn: DatabaseConnection, emitter: EventEmitter) -> Arc`。 +- `request_parse(&self, conversation_id: i32)`、`submit_turns(&self, conversation_id: i32, turns: Vec)`、`request_delete(&self, conversation_id: i32)`。 +- `IndexStatus { mode, indexed_conversations, total_conversations, building, progress }`。 + +- [ ] 先写队列去重和 diff 逻辑测试：同一会话重复投递只入队一次；内容哈希不变时不写；哈希变化时 upsert 并同步已启用的 FTS。 +- [ ] worker 内解析在 `spawn_blocking` 执行，解析器选择复用 `get_folder_conversation_core` 的 `match agent_type` 逻辑；默认最多 2 个并发，每个会话处理后 `tokio::task::yield_now().await`。 +- [ ] 启动时 `ensure_search_state`；状态缺失或 `schema_version` 落后时按 `updated_at DESC` 排队可见会话；每 10 分钟做一次漂移比较。 +- [ ] 在 `get_folder_conversation_with_live_core` 解析成功后投递已解析 turns；在导入新增/更新、TurnComplete 后投递 parse；删除路径先删文档再删 FTS。 +- [ ] 进度写回 `search_index_state` 并通过 `SEARCH_INDEX_PROGRESS_EVENT` 广播。 +- [ ] 桌面与服务器启动路径都持有同一个 `Arc`；Tauri 使用 `.manage()`，HTTP AppState 增加字段。 +- [ ] 运行 `cargo test --features test-utils search::indexer`、`cargo check --features test-utils`。 +- [ ] 提交：`feat(search): add background message indexer and hooks`。 + +## Task 6: 前端搜索接入与索引进度 + +**Files:** +- Modify: `src/lib/api.ts` +- Modify: `src/lib/tauri.ts` +- Modify: `src/lib/types.ts` +- Modify: `src/components/conversations/search-command-dialog.tsx` +- Modify: `src/contexts/search-dialog-context.tsx` 或新建 `src/hooks/use-search-index-status.ts` + +**Interfaces:** +- `searchConversations(params): Promise`。 +- `getSearchIndexStatus(): Promise`。 +- 对话框会话页签调用新接口，`CommandDialog shouldFilter={false}`；正文结果显示 `snippet_prefix + snippet_match + snippet_suffix`。 + +- [ ] 给对话框测试 mock `searchConversations`，先写“正文命中但标题不含关键词仍显示”的失败测试。 +- [ ] 实现接口与渲染，保持文件页签不变。 +- [ ] 新增进度 hook，打开搜索框时请求状态，订阅或 5 秒轮询，显示“索引中 x%”。 +- [ ] 运行 `pnpm test src/components/conversations/search-command-dialog.test.tsx` 和 `pnpm eslint .`。 +- [ ] 提交：`feat(search): show content matches and index progress in search dialog`。 + +## Task 7: 设置与国际化 + +**Files:** +- Modify: `src-tauri/src/commands/search.rs`（`get_search_settings_core`、`set_search_settings_core`，读写 `search_index_state.user_enabled/user_mode`） +- Modify: `src-tauri/src/web/handlers/search.rs` 与路由 +- Modify: `src/lib/api.ts`、`src/lib/tauri.ts`、`src/lib/types.ts` +- Modify: 设置页新增 `SearchSettingsSection`，调用上述命令，不依赖 localStorage +- Modify: `i18n/messages/en.json`、`zh-CN.json`、`zh-TW.json`、`ja.json`、`ko.json`、`es.json`、`de.json`、`fr.json`、`pt.json`、`ar.json` + +**Interfaces:** +- 设置值保存在 `search_index_state.user_enabled` 与 `user_mode`，`user_mode` 取值 `auto|scan|fts`；前端设置页展示一个开关和一个三选一模式。 +- 文案键 `Folder.search.indexing`、`Folder.search.contentMatch` 等。 + +- [ ] 先实现并测试两个设置命令，再写设置页；`user_enabled=0` 时 `search_conversations_core` 只返回标题结果，`user_mode` 覆盖自动阈值切换。 +- [ ] 补齐 10 种语言文案，中文为“正在建立内容索引”“正文匹配”。 +- [ ] 运行 `pnpm test` 相关设置测试和 `pnpm eslint .`。 +- [ ] 提交：`feat(search): add search settings and i18n`。 + +## Task 8: 基准、恢复校验与最终加固 + +**Files:** +- Create: `src-tauri/src/search/bench.rs` 或 `src-tauri/tests/search_perf.rs` +- Modify: `src-tauri/src/db/mod.rs`（启动恢复校验钩子，如适用） + +**Interfaces:** +- 无新公共接口，仅测试和启动校验。 + +- [ ] 构造 100k 轮次种子数据，测量扫描模式和 FTS 模式 p50/p95，记录文档表、trigram、短词表体积。 +- [ ] 添加恢复校验：文档数与可见会话数不一致时标记全量待建。 +- [ ] 运行 `pnpm test`、`cargo test --features test-utils`、`cargo clippy --all-targets --features test-utils -- -D warnings`、`cargo check --no-default-features --bin codeg-server`。 +- [ ] 提交：`test(search): add perf benchmarks and recovery checks`。 + + +## 实现差异记录（评审后修订） + +- 短词表改为 `detail=column`，列限定 MATCH（`words :` / `bigrams :`）可用；升级由 schema v3 在后台重建。 +- 放弃 contentless 布局下的 bm25 排序，改为 `updated_at DESC, id DESC` 的确定性排序，多词查询仍取交集。 +- 内容查询的文件夹 / agent 过滤写进候选 SQL，`LIMIT` 在可见性过滤之后生效。 +- 模式切换与 FTS 重建合并在同一事务，失败回滚并可在下个 tick 重试；设置保存只落库，重建交给后台 worker。 +- `scan_ms_per_mb` / `last_calibration_at` / `short_threshold_mb` 三个从未读写的列已删除，对应校准 / 看门狗设计不再保留。 +- 关闭“内容搜索”时清空已存储的正文文档，避免开关语义变成“仍在保存我的正文”。 +- 备份不包含会话内容时，从 `VACUUM INTO` 快照中剥离搜索文档与 FTS 表。 +- 命中定位改用后端字符偏移 + CSS Custom Highlight API，不再改写 React 托管的 DOM；多词查询和合并 turn 的计数与高亮对齐。 +- 单次查询长度上限 256 字符；漂移核对改为单条 LEFT JOIN；索引器待处理集合修复为批处理结算。 diff --git a/docs/specs/2026-08-14-message-content-search-design.md b/docs/specs/2026-08-14-message-content-search-design.md new file mode 100644 index 000000000..92f22d351 --- /dev/null +++ b/docs/specs/2026-08-14-message-content-search-design.md @@ -0,0 +1,427 @@ +# 会话消息内容搜索设计规格 + +Status: 待评审，评审通过后进入实现 + +Date: 2026-08-14 + +Branch: `task/1` + +## 1. 背景与问题 + +工作区左侧的搜索功能由 `Ctrl+K` 打开。会话页签目前只按会话标题搜索： + +- 前端 [search-command-dialog.tsx](../../src/components/conversations/search-command-dialog.tsx) 调用 `list_all_conversations`，把用户输入作为 `search` 参数传给后端。 +- 后端 `conversation_service::list_all` 只执行 `conversation::Column::Title.contains(s)`，因此聊天正文中的关键词不会被命中。 + +目标是在不降低现有搜索响应速度、不明显增加存储和 CPU 的前提下，把会话正文纳入搜索结果。正文当前不保存在 codeg 数据库中，而是在打开会话时由各 Agent 解析器从转录文件读取，所以本设计同时解决“何时解析正文”和“如何快速查询正文”两个问题。 + +## 2. 目标与非目标 + +目标： + +- 支持按用户消息和助手回复的纯文本内容搜索会话。 +- 保持搜索框 300ms 防抖体验，内容搜索后端查询目标 p95 不超过 45ms。 +- 首次建索引在应用启动后自动后台完成，搜索路径永不触发转录文件解析。 +- 尽量控制索引体积，提供扫描模式与 FTS 模式的自适应切换。 +- 桌面模式和服务器模式共用同一套核心逻辑。 + +非目标： + +- 不索引 `Thinking`、`System`、工具调用、工具结果、图片和文件内容。 +- 不改动“文件”页签，该页签仍只匹配文件名与路径。 +- 不做语义搜索、向量搜索或远程搜索服务。 +- 不改变既有 `list_all_conversations` 的标题搜索接口行为。 + +## 3. 已确认决策 + +1. 索引范围仅限 `MessageTurn` 中 `role` 为 `User` 或 `Assistant` 的 `ContentBlock::Text` 文本。 +2. 搜索范围与现在一致：选中文件夹时只搜该文件夹；无文件夹时搜全部非删除文件夹，并保留 Agent 过滤。 +3. 首次建索引指升级到该功能版本后的第一次启动。启动检测到未建索引后立即后台静默执行，后续启动只做增量同步。 +4. 排序：标题命中优先；内容命中按相关度，相同相关度按 `updated_at` 倒序。 +5. 自适应阈值默认 40MB，当前机器实测的 29MB 可索引文本停留在扫描模式。 + +## 4. 现状与实测基线 + +在本机对可发现的 Agent 会话做抽样和全量统计，结果如下： + +| 来源 | 原始会话体积 | 用户 + 助手纯文本 | +| --- | ---: | ---: | +| Codex | 4,266MB，2,314 个文件 | 约 25.0MB，95% 区间 17.5 到 35.2MB | +| Claude | 92.5MB，51 个文件 | 约 1.5MB，区间 1.0 到 2.0MB | +| OpenCode | 832MB SQLite 数据库 | 1.6MB，全量统计 | +| Pi | 15.0MB，24 个文件 | 0.8MB，全量统计 | +| 合计 | 约 5.2GB | 约 29MB，区间约 21 到 40MB | + +Codex 抽样 60 个文件共 1.09GB，按文件字节加权；Claude 抽样 20 个文件；OpenCode 和 Pi 全量解析。每条文本按 8KB 截断。 + +SQLite FTS5 实测，样本 9.55 到 9.67MB 文本： + +| 布局 | 相对可索引文本 | +| --- | ---: | +| 每条消息一行保存原文 | 1.15 倍 | +| 每个会话一行保存原文 | 1.02 倍 | +| trigram 倒排，contentless + `detail=none` | 1.34 倍 | +| 短词倒排，contentless unigram + bigram | 1.26 倍 | +| 会话行 + trigram 合计 | 2.36 倍 | + +结论：会话行布局下，29MB 文本在扫描模式约占 30MB；开启 trigram 后约占 68MB，95% 区间约 50 到 95MB；若大语料下再启用短词索引，总额外约 3.6 倍。 + +扫描最坏耗时实测（热缓存，统一采用无匹配查询强制扫完全表；罕见命中会在此基础再增加数毫秒）： + +| 文本 | p50 | p95 | +| --- | ---: | ---: | +| 10MB | 8ms | 9ms | +| 29MB | 25ms | 26ms | +| 100MB | 85ms | 87ms | +| 200MB | 178ms | 185ms | + +因此 150 到 200MB 阈值过晚，本设计采用 40MB 默认阈值。 + +## 5. 架构总览 + +新增模块： + +- `MessageSearchNormalizer`：从已解析的 `MessageTurn` 提取规范化文本并计算哈希。 +- `MessageSearchDocumentService`：维护每会话一行原文文档。 +- `MessageSearchIndexer`：后台索引 worker，负责首建、增量、漂移扫描和模式切换。 +- `MessageSearchService`：合并标题与正文结果，生成排序和摘要片段。 +- `MessageSearchState`：保存模式、阈值、校准值和进度。 + +数据流： + +1. 会话打开时，`get_folder_conversation_core` 已经解析出全部轮次；规范化后的文档异步投递到 worker，不重复读取转录文件。 +2. 导入、live turn 完成、会话删除等事件同样投递 worker。 +3. worker 在后台按会话做 diff，只写新增或变化的文档。 +4. Ctrl+K 搜索只读 SQLite：标题查询走现有服务，正文查询走文档表或 trigram 表。 +5. 可索引文本超过阈值时，worker 后台构建 trigram 表，完成后原子切换到 FTS 模式；短词查询持续超预算时再构建短词倒排。 + +## 6. 数据模型与迁移 + +### 6.1 文档表 + +```sql +CREATE TABLE message_search_document ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + conversation_id INTEGER NOT NULL, + text TEXT NOT NULL, + content_hash TEXT NOT NULL, + source_ended_at TEXT, + source_message_count INTEGER NOT NULL DEFAULT 0, + updated_at TEXT NOT NULL +); + +CREATE UNIQUE INDEX idx_message_search_document_conversation + ON message_search_document(conversation_id); +``` + +`id` 同时是 trigram 和短词表的 rowid。更新文档时保留 `id`，避免两个 FTS rowid 漂移；若实现选择删除重建，则必须先删除旧 FTS rowid。`content_hash` 是规范化文本的 SHA-256 十六进制值，用于 diff。`source_ended_at` 与 `source_message_count` 来自解析器列表摘要，用于漂移检测。 + +### 6.2 trigram 表 + +迁移时创建空表，不插入任何行： + +```sql +CREATE VIRTUAL TABLE message_search_trigram USING fts5( + text, + content='', + contentless_delete=1, + detail=none, + tokenize='trigram' +); +``` + +空表只有少量页开销。扫描模式下保持为空；FTS 模式下 `rowid` 与 `message_search_document.id` 一一对应。 + +### 6.3 短词表 + +短词表在迁移时创建为空表，默认不填充，仅在短词扫描持续超过延迟预算时写入： + +```sql +CREATE VIRTUAL TABLE message_search_short USING fts5( + words, + bigrams, + content='', + contentless_delete=1, + detail=none, + tokenize='unicode61 remove_diacritics 2' +); +``` + +`words` 保存拉丁小写单词和中文单字，`bigrams` 保存中文连续二字组，两者都以空格分隔。`rowid` 同样与 `message_search_document.id` 一一对应。 + +### 6.4 状态表 + +```sql +CREATE TABLE search_index_state ( + id INTEGER PRIMARY KEY CHECK (id = 1), + schema_version INTEGER NOT NULL DEFAULT 1, + mode TEXT NOT NULL DEFAULT 'scan', + threshold_mb REAL NOT NULL DEFAULT 40.0, + short_fts_enabled INTEGER NOT NULL DEFAULT 0, + short_threshold_mb REAL NOT NULL DEFAULT 40.0, + scan_ms_per_mb REAL, + indexed_conversation_count INTEGER NOT NULL DEFAULT 0, + last_calibration_at TEXT, + last_backfill_at TEXT +); +``` + +首次启动时插入单行状态。schema 升级时递增 `schema_version` 并触发重扫。 + +### 6.5 迁移 + +在现有 `m20260807_000001_work_task_scheduled_at` 之后新增迁移。两个虚拟表用 raw SQL 创建。备份与恢复沿用 SQLite 全库备份，新表自动包含在内；恢复后启动校验文档数量与有 `external_id` 的可见会话数量，不一致则重新排队。 + +## 7. 文本规范化 + +对每个 `MessageTurn`： + +1. 仅接受 `role` 为 `User` 或 `Assistant`。 +2. 遍历 `blocks`，只保留 `ContentBlock::Text { text }`。 +3. 每个文本块先 `trim`，丢弃空串，UTF-8 字节超过 8,192 时截断到合法字符边界。 +4. 同一会话的所有块按原始顺序以 `\n\n` 连接。 +5. 对最终文本计算 SHA-256，十六进制小写保存为 `content_hash`。 + +系统注入和内部上下文在解析层即被排除，不依赖查询候选上限。Codex 的 `session_meta.base_instructions`、`response_item` 中 developer/system 及文本型 user 消息、`environment_context`、`turn_context`、reasoning 不进入 `MessageTurn` 的 `Text` 块；Claude、Pi、OpenCode 同样只取显式 user 或 assistant 的 `type: text` 块，忽略 thinking、attachment、tool 和 system 内容。 + +不主动转小写、不做分词。大小写无关由 SQLite `LIKE`（ASCII）和 trigram 折叠处理，中文无需大小写。已知边界：扫描模式的 `LIKE` 大小写折叠覆盖 ASCII；带重音的非 ASCII 字母的大小写变体仅在 FTS 模式由 trigram 的 Unicode 折叠覆盖。 + +## 8. 索引策略 + +### 8.1 两种模式 + +- `scan`：只维护 `message_search_document`，搜索使用参数化 `LIKE`。 +- `fts`：维护 `message_search_trigram`，3 个 Unicode 字符及以上的词走 FTS；1 到 2 字符的词先走文档表扫描，短词查询持续超预算后启用 `message_search_short`。 + +### 8.2 自适应阈值 + +默认阈值 `threshold_mb = 40.0`，延迟预算 `budget_ms = 45.0`。 + +首次全量回填完成后做本机校准： + +1. 对文档表执行 3 次无匹配 `LIKE` 查询，测量毫秒每 MB。 +2. 计算 `threshold_mb = clamp(budget_ms / measured_ms_per_mb, 24.0, 64.0)`。 +3. 校准失败时使用 40.0。 + +切换规则： + +- 当前可索引文本字节数达到阈值时，后台构建 trigram 表并切到 `fts`。 +- 可索引文本降到阈值的 50%（至少 16MB）以下时，删除 trigram 行并切回 `scan`，避免反复震荡。 +- 升级期间继续使用扫描路径，构建完成后原子更新 `mode`。 + +### 8.3 运行期看门狗 + +索引器按滚动窗口记录最近 30 次扫描查询耗时。每 10 次计算一次 p95；连续 3 个窗口超过 `budget_ms * 1.5` 时，即使未达到字节阈值也提前升级：若当前是扫描模式则构建 trigram，若瓶颈来自 1 到 2 字符查询则再构建短词索引。覆盖冷缓存和慢磁盘。 + +### 8.4 短词索引 + +trigram 无法加速 1 到 2 字符的子串查询，所以单独用短词索引兜底： + +1. 默认在可索引文本达到 `short_threshold_mb = 40.0` 时与 trigram 一起构建，保证 1 到 2 字符查询也满足 45ms 预算。 +2. 短词扫描看门狗也可提前触发：最近 30 次 1 到 2 字符查询的 p95 连续 3 个窗口超过 `budget_ms * 1.5` 时构建。 +3. 文本降到 `short_threshold_mb / 2` 以下且短词 p95 回落后，删除短词索引行并停用。 +4. 构建期间短词查询继续走文档表扫描，完成后原子更新 `short_fts_enabled`。 + +## 9. 查询算法 + +新增 `search_conversations_core`，输入 `folder_ids`、`agent_type`、`query`、`limit`，输出 `Vec`。 + +### 9.1 查询归一化 + +- `trim` 并折叠连续空白。 +- 按 Unicode 空白拆词，最多 8 个词。 +- 单次查询长度最多 256 个字符。 +- 空查询沿用现有最近会话列表行为。 + +### 9.2 候选召回 + +每个词独立查询，最终在 Rust 中按 `conversation_id` 求交集，允许多个词出现在同一会话的不同轮次。 + +扫描模式： + +```sql +SELECT conversation_id +FROM message_search_document +WHERE text LIKE ? ESCAPE '\'; +``` + +参数是转义 `%`、`_`、`\` 后的 `%term%`。该模式返回全部命中会话 id，候选集合最多等于可见会话数；相关位置在 Rust 对交集后的集合计算，避免提前截断造成漏召回。 + +FTS 模式，词长度至少为 3 时： + +1. 生成连续三字符片段，每段写成加双引号的 FTS 短语项。 +2. 各片段用 `AND` 连接，先取 rowid 候选。 +3. join 文档表后以 `text LIKE ?` 精确过滤，消除非相邻片段造成的假阳性。 +4. 使用 `bm25(message_search_trigram)` 作为相关度。 + +每词的候选上限为 `max(1, 当前查询范围内的可见会话数)`。可见会话数按 9.3 的过滤条件计算。索引一行对应一个会话，因此任何单个词最多只能命中可见会话数；该默认值保证不会静默漏召回。若未来可见会话数极大，可显式配置更小上限，但必须把“召回被截断”作为结果元数据返回。 + +词长度为 1 到 2 时，若 `short_fts_enabled = 1`： + +1. 中文单字查 `words` 列的精确项，二字中文按相邻二字组做短语查询。 +2. 拉丁词按 `words` 列前缀查询。 +3. join 文档表后以 `text LIKE ?` 精确过滤。 + +未启用短词索引时，短词与扫描模式相同。 + +### 9.3 过滤、合并与排序 + +候选 `conversation_id` 继续 join `conversation`，沿用现有可见性条件： + +- `deleted_at IS NULL` +- `kind != 'loop'` +- `parent_id IS NULL` +- 文件夹集合与 Agent 过滤 + +标题结果由现有 `list_all` 查询获取。合并时按会话去重，标题命中整体排前；内容命中按相关度，相关度相同按 `updated_at DESC`。返回上限 50。 + +扫描模式的相关度使用首个命中位置，位置越靠前越相关；多词时取各词位置之和。FTS 模式使用 `bm25`，多词交集取最差值，保持单调可比。 + +### 9.4 摘要片段 + +仅对最终返回的 50 条生成片段： + +- 在文档 `text` 中做大小写无关的首个命中定位；定位使用文档与查询词的 Unicode 小写折叠副本，只用于计算偏移，不修改存储文本。 +- 向前、向后各取约 80 个字符窗口，窗口边界不切断 UTF-8 字符。 +- 返回结构化字段 `snippet_prefix`、`snippet_match`、`snippet_suffix`，避免把高亮标记注入用户文本。 + +### 9.5 安全与注入 + +- `LIKE` 使用绑定参数并显式 `ESCAPE`。 +- FTS `MATCH` 只由本设计生成的三字符带引号项构成，不把原始输入拼进表达式；引号按 FTS5 规则加倍。 +- 所有查询有长度、词数和自适应召回上限，防止恶意超长输入。 + +## 10. 后台索引器 + +### 10.1 队列与去重 + +worker 使用有界队列，按 `conversation_id` 合并重复请求。解析在 `spawn_blocking` 中执行，复用与 `get_folder_conversation_core` 相同的 `AgentParser::get_conversation` 路径。 + +文档新增或内容哈希变化时，在同一事务内同步 upsert 已启用的 trigram 和短词 FTS rowid；未启用的表保持空。 + +### 10.2 首建与漂移 + +首次启动时： + +1. 若状态表的 `schema_version` 不存在或落后，标记全量待建。 +2. 按 `conversation.updated_at DESC` 排队所有满足可见性条件的会话：`external_id` 非空、`parent_id` 为空、`kind != 'loop'`、`deleted_at` 为空。 +3. 每个会话处理后主动让出一次，默认最多 2 个并发解析。 + +每 10 分钟运行一次漂移检查： + +1. 用现有解析器 `list_conversations` 得到廉价摘要。 +2. 与会话行及文档行的 `(external_id, source_ended_at, source_message_count)` 比较。 +3. 只有摘要变化的会话才排队重新解析。 + +### 10.3 事件钩子 + +- 会话详情加载：`get_folder_conversation_with_live_core` 解析完成后，把已有 `turns` 的规范化结果异步投递。 +- 导入：`import_summaries` / `import_summaries_resilient` 产生新增或更新时投递会话 id。 +- live turn 完成：ACP `TurnComplete` 之后投递会话 id，由 worker 重新解析已落盘的转录。 +- 会话软删除：同步删除文档行及 trigram、短词两个 FTS 表中对应的 rowid。 + +### 10.4 失败与进度 + +- 单个解析失败记录日志，保留待重试，不中断队列。 +- 重试采用退避，连续失败跳过该会话并计入状态。 +- 进度通过 `EventEmitter` 发送 `search_index_progress`，桌面模式走 Tauri 事件，服务器模式走 WebSocket。 +- 前台打开搜索框时显示“索引中 x%”，标题搜索始终可用。 + +## 11. API 与前端 + +### 11.1 共享模型 + +```rust +pub struct DbConversationSearchResult { + pub summary: DbConversationSummary, + pub match_kind: SearchMatchKind, // Title | Content | Both + pub snippet_prefix: Option, + pub snippet_match: Option, + pub snippet_suffix: Option, + pub content_match_count: u32, +} +``` + +新增 `search_conversations` 命令和 Axum handler，两者共享 `search_conversations_core`。原 `list_all_conversations` 保持不变。 + +### 11.2 前端改动 + +- `search-command-dialog.tsx` 会话页签改用 `searchConversations`。 +- `CommandDialog` 在会话页签设置 `shouldFilter={false}`，避免 cmdk 按标题二次过滤正文命中结果。 +- 每条结果显示标题、Agent、时间和一个不超过两行的正文摘要。 +- 订阅或轮询索引进度，在搜索框显示构建状态。 +- 新增设置：`content_search_enabled`、`content_search_mode`（`auto` | `scan` | `fts`）。 +- 为 10 种语言补充“正在建立内容索引”“正文匹配”等文案。 + +文件页签不改变。 + +## 12. 性能与存储验收 + +| 指标 | 目标 | +| --- | --- | +| 搜索路径转录文件 I/O | 0 | +| 扫描模式 p95 | 不超过 45ms，且不超过阈值文本量 | +| FTS 模式 3 字符以上 p95 | 不超过 15ms，100k 轮次 | +| FTS 模式短词索引后 1 到 2 字符 p95 | 不超过 15ms，100k 轮次 | +| 首建期间界面 | 无阻塞，后台 2 并发 | +| 文档表体积 | 可索引文本的 1.05 倍以内 | +| trigram 额外体积 | 可索引文本的 1.5 倍以内 | +| 短词索引额外体积 | 可索引文本的 1.5 倍以内 | +| 召回完整性 | 默认候选上限不低于可见会话数，不静默截断 | +| 删除一致性 | 会话删除后无孤儿索引行 | + +实现阶段用种子数据跑基准，达不到扫描 p95 目标时优先收紧阈值校准系数，而不是修改 45ms 预算。 + +## 13. 测试计划 + +后端单元测试： + +- 规范化只保留用户与助手文本、截断、哈希稳定。 +- `LIKE` 转义与 FTS 三字符生成，含引号和中文。 +- 扫描与 FTS 两种模式的召回、求交集、过滤、合并排序。 +- 短词索引的中文单字、二字短语和拉丁前缀查询。 +- 自适应召回上限在可见会话数变化时仍不漏命中，超大规模截断可观测。 +- 片段窗口与 UTF-8 边界。 +- 文档 diff、删除同步、软删除清理。 +- 阈值计算、滞后区间和看门狗状态机。 +- 迁移 up/down 和恢复后校验。 + +前端测试： + +- 正文命中结果不被 cmdk 过滤。 +- 摘要渲染和索引进度显示。 +- Agent 过滤与文件夹范围参数正确传递。 + +基准测试： + +- 构造 100k 轮次，分别断言扫描与 FTS 模式延迟。 +- 记录文档表和 FTS 表实际膨胀比。 + +## 14. 实施顺序 + +1. 数据模型迁移、状态表和空 FTS 表。 +2. 文本规范化与哈希模块，附单元测试。 +3. 文档 service 的 upsert、diff、删除。 +4. 后台索引器、事件钩子、进度事件。 +5. 查询 service、共享模型、Tauri 命令与 HTTP handler。 +6. 前端搜索框切换、摘要、进度和设置。 +7. 国际化文案。 +8. 性能基准、阈值校准和整体验收。 +9. 4 到 5GB 现有会话的后台回填实测。 + +## 15. 风险与回滚 + +- FTS 表损坏或构建失败：保持扫描模式，记录错误，后台自动重建。 +- 短词索引损坏或构建失败：短词回退到文档表扫描，自动重建。 +- 首次回填的磁盘和 CPU：限并发与主动让出，进度可见，可暂停。 +- 模式切换期间的查询：原子更新 `mode`，切换前后都可用。 +- 数据库写入竞争：每会话小事务，搜索路径只读，WAL 保持读不阻塞。 +- 回滚：新表不改变既有标题搜索，回滚该功能只删除新表和设置即可。 + +## 16. 被否方案 + +- 实时 grep 转录文件加缓存：冷查询延迟不可控。 +- Tantivy 或 Meilisearch：能力强但引入外部索引文件和进程，备份与部署复杂度高。 +- `words + bigrams + trigram` 双 FTS：实测膨胀到可索引文本约 10 倍，改为自适应单 trigram。 +- 固定 2,000 候选上限：会造成静默漏召回且易与系统提示词排除混淆，改为按可见会话数自适应。 +- 搜索时才首次建索引：会引入等待，已改为启动即后台首建。 diff --git a/docs/specs/2026-08-15-search-result-jump-highlight-design.md b/docs/specs/2026-08-15-search-result-jump-highlight-design.md new file mode 100644 index 000000000..24b119421 --- /dev/null +++ b/docs/specs/2026-08-15-search-result-jump-highlight-design.md @@ -0,0 +1,227 @@ +# 搜索命中跳转与闪烁高亮设计 + +## 背景 + +当前 Ctrl+K 搜索已经能在会话标题和正文中找到关键词。用户点击搜索结果后进入会话时，还看不到关键词在哪条消息中，也无法快速定位。 + +本次新增交互目标： + +1. 点击搜索结果后自动滚动到第一条命中的正文位置，并让该位置停在屏幕中部附近。 +2. 命中的关键词以黄色闪烁后渐隐。 +3. 如果命中的是标题，则高亮标题中的关键词并闪烁。 +4. 如果正文有多条命中，提供“下一条匹配”按钮，支持循环跳转。 + +## 范围 + +- 只针对 Ctrl+K 会话搜索结果进入会话后的跳转。 +- 不改变现有标题搜索、正文搜索、文件搜索和索引策略。 +- 不持久保存高亮状态，高亮动画结束后恢复普通文本。 +- 正文匹配只处理 User/Assistant 文本块；工具、图片、系统提示、思考内容不参与。 + +## 交互选择 + +用户已确认选择方案 B： + +- 自动定位并闪烁第一条正文命中； +- 显示“下一条匹配”按钮； +- 多条命中按会话顺序循环跳转； +- 标题命中直接高亮标题，不显示“下一条”。 + +## 架构 + +### 后端 + +新增精确匹配位置模型，并返回给前端。 + +#### 1. 规范化文本块偏移表 + +`message_search_document` 增加一个 `block_offsets` JSON 字段。每个索引文本块保存： + +```json +[ + { + "turn_id": "turn-01", + "block_index": 0, + "start": 0, + "end": 120, + "leading_trim": 3 + } +] +``` + +字段含义： + +- `turn_id`：后端 `MessageTurn.id`。 +- `block_index`：该 turn 中的 Text block 索引。 +- `start` / `end`：在规范化全文中的字符区间，半开区间。 +- `leading_trim`：原始 block 文本开头被 `trim()` 去掉的字符数，用于把规范化区间映射回原始 block。 + +### 2. 搜索结果模型 + +`DbConversationSearchResult` 增加： + +```rust +pub matches: Vec, +pub total_match_count: u32, +``` + +`SearchMatchLocation`： + +```rust +pub enum SearchMatchLocationKind { + Title, + Content, +} + +pub struct SearchMatchLocation { + pub kind: SearchMatchLocationKind, + pub turn_id: Option, + pub block_index: Option, + pub char_start: usize, + pub char_end: usize, +} +``` + +排序规则： + +- 标题命中排在最前。 +- 正文命中按 `turn_id` 在会话中的顺序排列；同一 turn 中按 `block_index` 和字符位置排序。 +- 结果最多返回 200 条匹配位置，避免极端大文档导致响应膨胀。 +- 标题命中只包含 `kind=title`，不包含 `turn_id`/`block_index`。 + +### 3. 查询路径 + +- 搜索服务读取最终结果对应的 `message_search_document`，使用 `block_offsets` 和规范化文本计算所有命中位置。 +- 查询路径不读取原始转录文件，保持既有约束。 +- `snippet_prefix/match/suffix` 继续由第一条内容命中生成。 +- `content_match_count` 改为实际统计值，但显示和返回仍可受上限保护。 + +### 4. 索引与迁移 + +- 在现有迁移之后新增一个小迁移，给 `message_search_document` 增加 `block_offsets` 字段。 +- 将 `search_index_state.schema_version` 提升到 2，触发已有索引重新构建。 +- 索引器在 `normalize_turns` 时同时生成 block offset 清单，并在 upsert 文档时写入。 +- 旧文档如果没有 `block_offsets`，搜索时降级为“只返回 snippet，不返回精确跳转”，前端只显示无跳转结果，避免破坏搜索。 + +### 前端 + +#### 1. 搜索焦点状态 + +新增一个轻量级搜索焦点状态，保存从搜索结果到会话页的传递信息： + +```ts +interface SearchFocus { + conversationId: number + query: string + matches: SearchMatchLocation[] + activeMatchIndex: number +} +``` + +搜索对话框选择会话时： + +1. 把 `SearchFocus` 写入状态； +2. 照常调用 `openTab`； +3. 对话框关闭。 + +状态不写入持久化存储。 + +#### 2. 会话详情加载与跳转 + +`ConversationDetailPanel` / `MessageListView` 读取当前会话的 `SearchFocus`： + +- 等待 detail 加载完成。 +- 根据 `turn_id` 找到对应 timeline item 的索引。 +- 如果目标 turn 不在当前加载窗口： + - 调用 `loadOlderTurns`； + - 最多尝试固定次数； + - 超过上限则停止并清除搜索焦点。 +- 使用虚拟列表的 `scrollApiRef.scrollToIndex(index, { align: "center" })` 或等价接口滚动。 +- 滚动完成后再触发高亮。 + +#### 3. 正文高亮 + +- 只对当前激活匹配对应的 turn/block 做高亮。 +- 高亮操作在渲染后的 DOM 文本节点中进行： + - 找到目标 block 的文本范围； + - 用 `TreeWalker` / 文本节点扫描定位关键词； + - 把命中的文本节点拆开并包成 ``。 +- 不修改 Markdown 解析器和内容数据，避免影响复制、导出、编辑和后续流式更新。 +- 动画结束或切换到下一条时移除旧 ``。 + +#### 4. 下一条匹配 + +- 仅当正文匹配数量大于 1 时显示按钮。 +- 按钮放在消息列表右侧或滚动按钮附近。 +- 点击后： + - `activeMatchIndex = (activeMatchIndex + 1) % matches.length`； + - 滚动到新位置； + - 重新闪烁。 +- 按钮显示当前序号和总匹配数，例如 `2 / 8`。 + +#### 5. 标题命中 + +- 标题命中 `turn_id` 为空。 +- `ConversationDetailHeader` 根据 `char_start` / `char_end` 将标题子串包成 ``。 +- 使用与正文相同的闪烁动画。 +- 标题命中不触发消息列表滚动，也不显示“下一条”。 + +#### 6. 动画 + +CSS keyframes： + +```css +@keyframes search-flash { + 0% { background-color: #fde047; } + 100% { background-color: transparent; } +} + +[data-search-flash] { + animation: search-flash 1.8s ease-out forwards; +} +``` + +动画结束后： + +- 移除 `` 或将其样式恢复为透明； +- 不残留黄色背景。 + +## 错误处理 + +- 会话已删除：清除 `SearchFocus`，不阻止会话打开。 +- 索引状态落后或匹配位置不可用：只滚动到对应消息，降级为不高亮。 +- 找不到对应 DOM 文本：不清除会话状态，只跳过闪烁。 +- 多条匹配中存在已被流式内容替换的位置：重新按当前 DOM 扫描一次；失败则跳到下一条。 + +## 兼容性 + +- 没有 `block_offsets` 的旧文档仍可被搜索，只是不能精确跳转。 +- 关闭正文搜索时不会产生正文匹配。 +- 不改变文件搜索、空查询、文件夹和 Agent 过滤行为。 +- Web 客户端和桌面客户端共用同一套数据结构和前端状态。 + +## 测试计划 + +### Rust + +- `block_offsets` JSON 序列化/反序列化。 +- Unicode 字符区间、`leading_trim` 映射。 +- 标题命中和正文命中排序。 +- 同一 turn 多个 block、同一 block 多次命中。 +- 旧文档缺少 offset 时的降级。 + +### 前端 + +- 选择搜索结果后写入 `SearchFocus`。 +- 等待 detail 后调用滚动接口。 +- “下一条”按钮循环和计数。 +- 正文 `mark` 动画类添加和移除。 +- 标题命中的 `mark` 渲染。 +- 找不到目标时的降级行为。 + +## 不做什么 + +- 不做跨会话的全局高亮。 +- 不做关键词在多个消息中的“全部标黄”。 +- 不做持久化搜索焦点。 +- 不改变搜索排序和 50 条上限。 diff --git a/src-tauri/src/app_state.rs b/src-tauri/src/app_state.rs index 7777e9c0e..65ec0c91f 100644 --- a/src-tauri/src/app_state.rs +++ b/src-tauri/src/app_state.rs @@ -75,6 +75,9 @@ pub struct AppState { /// The upgrade UI subscribes to it and re-syncs from a snapshot on mount, /// so download progress survives settings-page navigation and reloads. pub update_state: crate::update::AppUpdateStateHandle, + /// Background content-search indexer. Optional in tests and in + /// desktop-mode HTTP routers that cannot see a Tauri-managed instance. + pub search_indexer: Option>, } pub fn default_system_op_lock() -> Arc> { @@ -247,6 +250,7 @@ impl AppState { chat_authoring_config, system_op_lock: default_system_op_lock(), update_state: default_update_state(), + search_indexer: None, } } } diff --git a/src-tauri/src/bin/codeg_server.rs b/src-tauri/src/bin/codeg_server.rs index 7bff2ede8..afb088206 100644 --- a/src-tauri/src/bin/codeg_server.rs +++ b/src-tauri/src/bin/codeg_server.rs @@ -253,6 +253,8 @@ async fn async_main() -> ExitCode { event_bus_metrics.clone(), )); let emitter = EventEmitter::web_only(broadcaster.clone(), acp_event_bus.clone()); + let search_indexer = + codeg_lib::search::indexer::MessageSearchIndexer::spawn(db.conn.clone(), emitter.clone()); // Build AppState let pet_state_handle = codeg_lib::pet_state_mapper::new_pet_state_handle(); @@ -293,6 +295,7 @@ async fn async_main() -> ExitCode { chat_authoring_config: chat_authoring_config.clone(), system_op_lock: codeg_lib::app_state::default_system_op_lock(), update_state: codeg_lib::app_state::default_update_state(), + search_indexer: Some(search_indexer), }); // Logging phase 3: wire the emitter so the Logs viewer's live tail diff --git a/src-tauri/src/commands/backup/core.rs b/src-tauri/src/commands/backup/core.rs index d915b3342..e9d1a2a00 100644 --- a/src-tauri/src/commands/backup/core.rs +++ b/src-tauri/src/commands/backup/core.rs @@ -8,7 +8,7 @@ use std::path::{Path, PathBuf}; use chrono::Utc; -use sea_orm::{ConnectionTrait, DatabaseConnection, DbBackend, Statement}; +use sea_orm::{ConnectionTrait, Database, DatabaseConnection, DbBackend, Statement}; use sea_orm_migration::MigratorTrait; use tokio_util::sync::CancellationToken; @@ -65,6 +65,9 @@ pub(crate) async fn create_backup_core( return Err(cancelled_error()); } snapshot_db_to(inputs.conn, &db_snapshot).await?; + if !options.include_external_transcripts { + strip_search_content(&db_snapshot).await?; + } // ── Phase 2: build the ZIP payload (blocking) ──────────────────────── let manifest_template = BackupManifest { @@ -221,6 +224,31 @@ pub(crate) async fn scan_external_conflicts_core( .map_err(|e| AppCommandError::task_execution_failed("Scan task failed").with_detail(e.to_string()))? } +/// Remove indexed conversation text from a snapshot when the backup was +/// requested without content. `VACUUM INTO` copies the whole DB, so without +/// this pass the content search documents would ride along and silently defeat +/// the "include conversation content" switch. +async fn strip_search_content(snapshot: &Path) -> Result<(), AppCommandError> { + let url = format!("sqlite:{}?mode=rw", snapshot.to_string_lossy()); + let conn = Database::connect(url) + .await + .map_err(|e| AppCommandError::database_error("open backup snapshot").with_detail(e.to_string()))?; + for sql in [ + "DELETE FROM message_search_trigram", + "DELETE FROM message_search_short", + "DELETE FROM message_search_document", + "UPDATE search_index_state SET indexed_conversation_count = 0 WHERE id = 1", + ] { + conn.execute(Statement::from_string(DbBackend::Sqlite, sql.to_string())) + .await + .map_err(|e| { + AppCommandError::database_error("strip search content from snapshot") + .with_detail(e.to_string()) + })?; + } + Ok(()) +} + /// Run `VACUUM INTO` to produce a transactionally-consistent, defragmented /// single-file copy of the live DB — sidesteps the WAL `-wal`/`-shm` sidecars. pub(crate) async fn snapshot_db_to( diff --git a/src-tauri/src/commands/conversations.rs b/src-tauri/src/commands/conversations.rs index 2e793aef3..0e9a89853 100644 --- a/src-tauri/src/commands/conversations.rs +++ b/src-tauri/src/commands/conversations.rs @@ -457,7 +457,7 @@ pub async fn import_local_conversations_core( emitter: &EventEmitter, chat_channel_manager: &crate::chat_channel::manager::ChatChannelManager, folder_id: i32, -) -> Result { +) -> Result<(ImportResult, Vec), AppCommandError> { // Share IMPORT_GUARD with the batch importer: `(external_id, agent_type)` // has no DB unique index, so this legacy path racing a batch import (or a // second legacy call) could double-insert. try_lock rejects the overlap @@ -482,14 +482,19 @@ pub async fn import_local_conversations_core( // Broadcast a sidebar upsert for every title refreshed in place, so other // windows and web clients converge live, and propagate the new name to any - // bound chat thread — the same treatment the scan and list paths give a - // title discovered outside codeg. The importing client refetches the list - // itself, which also covers the newly imported rows. + // bound chat thread. The importing client refetches the list itself, which + // also covers the newly imported rows. drop( - notify_conversation_title_updates(conn, emitter, chat_channel_manager, updated_ids).await, + notify_conversation_title_updates( + conn, + emitter, + chat_channel_manager, + updated_ids.clone(), + ) + .await, ); - Ok(result) + Ok((result, updated_ids)) } #[cfg(feature = "tauri-runtime")] @@ -500,13 +505,24 @@ pub async fn import_local_conversations( chat_channel_manager: tauri::State<'_, crate::chat_channel::manager::ChatChannelManager>, folder_id: i32, ) -> Result { - import_local_conversations_core( + let (result, updated_ids) = import_local_conversations_core( &db.conn, - &EventEmitter::Tauri(app), + &EventEmitter::Tauri(app.clone()), &chat_channel_manager, folder_id, ) - .await + .await?; + { + use tauri::Manager; + if let Some(indexer) = + app.try_state::>() + { + for id in updated_ids { + indexer.request_parse(id); + } + } + } + Ok(result) } /// Serializes concurrent batch imports: `(external_id, agent_type)` has no DB @@ -693,8 +709,7 @@ pub async fn scan_importable_sessions_core( conn: &sea_orm::DatabaseConnection, emitter: &EventEmitter, chat_channel_manager: &crate::chat_channel::manager::ChatChannelManager, -) -> Result { - let progress_emitter = emitter.clone(); +) -> Result<(ScanResult, Vec), AppCommandError> { let progress_emitter = emitter.clone(); let summaries = import_service::collect_local_summaries(move |agent_type, done, total, session_count| { emit_event( @@ -721,7 +736,7 @@ async fn scan_importable_sessions_from_summaries( emitter: &EventEmitter, chat_channel_manager: &crate::chat_channel::manager::ChatChannelManager, summaries: Vec<(AgentType, ConversationSummary)>, -) -> Result { +) -> Result<(ScanResult, Vec), AppCommandError> { use sea_orm::{ColumnTrait, EntityTrait, QueryFilter}; let conv_rows = conversation::Entity::find() @@ -745,18 +760,23 @@ async fn scan_importable_sessions_from_summaries( // Refresh the already-imported rows in place before answering, then // broadcast each one so open sidebars re-sort without a refetch. The // chat-channel half runs detached so the scan never waits on Telegram. + let refreshed_ids = + import_service::sync_imported_sessions(conn, &conv_rows, &summaries).await; drop( notify_conversation_title_updates( conn, emitter, chat_channel_manager, - import_service::sync_imported_sessions(conn, &conv_rows, &summaries).await, + refreshed_ids.clone(), ) .await, ); let folder_rows = load_folder_rows(conn).await?; - Ok(build_scan_result(summaries, &imported_index, &folder_rows)) + Ok(( + build_scan_result(summaries, &imported_index, &folder_rows), + refreshed_ids, + )) } #[cfg(feature = "tauri-runtime")] @@ -766,7 +786,23 @@ pub async fn scan_importable_sessions( db: tauri::State<'_, AppDatabase>, chat_channel_manager: tauri::State<'_, crate::chat_channel::manager::ChatChannelManager>, ) -> Result { - scan_importable_sessions_core(&db.conn, &EventEmitter::Tauri(app), &chat_channel_manager).await + let (result, refreshed_ids) = scan_importable_sessions_core( + &db.conn, + &EventEmitter::Tauri(app.clone()), + &chat_channel_manager, + ) + .await?; + { + use tauri::Manager; + if let Some(indexer) = + app.try_state::>() + { + for id in refreshed_ids { + indexer.request_parse(id); + } + } + } + Ok(result) } /// Batch-import the selected sessions, creating (or reopening) each target @@ -1440,7 +1476,7 @@ pub fn resolve_turn_window_req( /// full turn list (delegation meta, auto-title, in-flight stamping) — slicing /// is strictly a serialization concern, so the windowed `turns` are identical /// to the corresponding region of the full response. -fn apply_turn_window( +pub(crate) fn apply_turn_window( detail: &mut DbConversationDetail, req: crate::commands::turn_window::TurnWindowReq, ) { @@ -1463,16 +1499,17 @@ fn apply_turn_window( /// reply persisted after it mid-stream. A no-op (one cheap lock pass) when no turn /// is in flight. Shared by the Tauri command and the web handler. /// -/// `window`: when set, the response's `turns` are sliced to the requested -/// window AFTER all full-list post-processing (the summary counts, stats and -/// watermark keep describing the full transcript). +/// This core function intentionally returns the FULL turn list. Callers that +/// need a window must call [`apply_turn_window`] themselves AFTER handing the +/// full turns to the search indexer — a windowed slice must never be written +/// into the content index, or opening a conversation would silently truncate +/// its indexed history. pub async fn get_folder_conversation_with_live_core( conn: &sea_orm::DatabaseConnection, manager: &crate::acp::manager::ConnectionManager, chat_channel_manager: &crate::chat_channel::manager::ChatChannelManager, emitter: &EventEmitter, conversation_id: i32, - window: Option, ) -> Result { let (mut detail, parsed_title) = get_folder_conversation_core(conn, conversation_id).await?; @@ -1515,9 +1552,6 @@ pub async fn get_folder_conversation_with_live_core( detail.in_flight_user_turn_id = apply_in_flight_message_id(&mut detail.turns, &pending, started_at); } - if let Some(req) = window { - apply_turn_window(&mut detail, req); - } Ok(detail) } @@ -1560,15 +1594,26 @@ pub async fn get_folder_conversation( from_index: Option, ) -> Result { let window = resolve_turn_window_req(tail_turns, from_index)?; - get_folder_conversation_with_live_core( + let mut result = get_folder_conversation_with_live_core( &db.conn, &manager, &chat_channel_manager, - &EventEmitter::Tauri(app), + &EventEmitter::Tauri(app.clone()), conversation_id, - window, ) - .await + .await?; + { + use tauri::Manager; + if let Some(indexer) = + app.try_state::>() + { + indexer.submit_turns(conversation_id, std::sync::Arc::new(result.turns.clone())); + } + } + if let Some(req) = window { + apply_turn_window(&mut result, req); + } + Ok(result) } #[cfg(feature = "tauri-runtime")] @@ -2316,8 +2361,17 @@ pub async fn delete_conversation( db: tauri::State<'_, AppDatabase>, conversation_id: i32, ) -> Result<(), AppCommandError> { - let emitter = EventEmitter::Tauri(app); - delete_conversation_with_cleanup_core(&emitter, &db.conn, conversation_id).await + let emitter = EventEmitter::Tauri(app.clone()); + delete_conversation_with_cleanup_core(&emitter, &db.conn, conversation_id).await?; + { + use tauri::Manager; + if let Some(indexer) = + app.try_state::>() + { + indexer.request_delete(conversation_id); + } + } + Ok(()) } fn compute_stats(all_conversations: &[ConversationSummary]) -> AgentStats { @@ -3704,7 +3758,7 @@ mod tests { ); summary.1.title = Some("Codex index title".into()); - let result = scan_importable_sessions_from_summaries( + let (result, _refreshed_ids) = scan_importable_sessions_from_summaries( &db.conn, &emitter, &chat_channel_manager, diff --git a/src-tauri/src/commands/mod.rs b/src-tauri/src/commands/mod.rs index fbf421909..853a1526a 100644 --- a/src-tauri/src/commands/mod.rs +++ b/src-tauri/src/commands/mod.rs @@ -32,6 +32,7 @@ pub mod remote_proxy; #[cfg(feature = "tauri-runtime")] pub mod remote_workspace; pub mod science; +pub mod search; pub mod session_info; pub mod system_settings; pub mod terminal; diff --git a/src-tauri/src/commands/search.rs b/src-tauri/src/commands/search.rs new file mode 100644 index 000000000..30dc93402 --- /dev/null +++ b/src-tauri/src/commands/search.rs @@ -0,0 +1,911 @@ +use std::collections::{HashMap, HashSet}; + +use sea_orm::{ConnectionTrait, DatabaseConnection, DbBackend, Statement}; + +use crate::app_error::AppCommandError; +use crate::db::error::DbError; +use crate::db::service::{ + conversation_service, message_search_service, + message_search_service::{MODE_FTS, MODE_SCAN, USER_MODE_FTS, USER_MODE_SCAN}, +}; +use crate::models::{ + AgentType, DbConversationSearchResult, DbConversationSummary, SearchIndexStatus, + SearchMatchLocation, SearchMatchLocationKind, +}; +use crate::search::normalizer::NormalizedBlockOffset; +use crate::search::query::{self, ShortTermQuery}; + +const DEFAULT_LIMIT: u64 = 50; +const SNIPPET_CONTEXT_CHARS: usize = 80; +const MAX_MATCH_LOCATIONS: usize = 200; +const MAX_QUERY_CHARS: usize = 256; + +pub async fn search_conversations_core( + conn: &DatabaseConnection, + folder_ids: Option>, + agent_type: Option, + query: String, + limit: Option, +) -> Result, AppCommandError> { + let limit = limit.unwrap_or(DEFAULT_LIMIT).clamp(1, 200); + let state = message_search_service::ensure_search_state(conn).await?; + let mut query = query.trim().to_string(); + // A pasted error message is a common content-search query; cap it so the + // cost stays bounded instead of building one LIKE pattern and gram per + // character. + if query.chars().count() > MAX_QUERY_CHARS { + query = query.chars().take(MAX_QUERY_CHARS).collect(); + } + + let title_summaries = conversation_service::list_all( + conn, + folder_ids.clone(), + agent_type, + if query.is_empty() { + None + } else { + Some(query.clone()) + }, + None, + None, + false, + ) + .await?; + + let terms = query::split_terms(&query); + if query.is_empty() || !state.user_enabled { + return Ok(title_only_results(title_summaries, &terms, limit as usize)); + } + + let effective_mode = match state.user_mode.as_str() { + USER_MODE_SCAN => MODE_SCAN, + USER_MODE_FTS => MODE_FTS, + _ => state.mode.as_str(), + }; + let scope = Scope::build(folder_ids.clone(), agent_type)?; + let cap = scope.count(conn).await?; + + let mut candidate_ids: Option> = None; + let mut updated_at: HashMap> = HashMap::new(); + for term in &terms { + let rows = term_candidates(conn, term, effective_mode, state.short_fts_enabled, &scope, cap) + .await?; + let ids: HashSet = rows.iter().map(|(id, _)| *id).collect(); + for (id, at) in rows { + updated_at.entry(id).or_insert(at); + } + candidate_ids = Some(match candidate_ids { + None => ids, + Some(mut existing) => { + existing.retain(|id| ids.contains(id)); + existing + } + }); + } + + let candidate_set = candidate_ids.unwrap_or_default(); + if candidate_set.is_empty() { + return Ok(title_only_results(title_summaries, &terms, limit as usize)); + } + + let mut title_results: Vec = + Vec::with_capacity(limit as usize); + let mut title_ids = HashSet::new(); + let mut needed_document_ids = HashSet::new(); + for summary in title_summaries.into_iter().take(limit as usize) { + let summary_id = summary.id; + title_ids.insert(summary_id); + if candidate_set.contains(&summary_id) { + needed_document_ids.insert(summary_id); + } + title_results.push(DbConversationSearchResult { + summary, + snippet_prefix: None, + snippet_match: None, + snippet_suffix: None, + matches: Vec::new(), + }); + } + + let remaining = limit as usize - title_results.len(); + let mut content_pick_ids = Vec::with_capacity(remaining); + let mut candidates_sorted: Vec = candidate_set.iter().copied().collect(); + candidates_sorted.sort_by(|a, b| { + updated_at + .get(b) + .cmp(&updated_at.get(a)) + .then_with(|| b.cmp(a)) + }); + for id in candidates_sorted { + if title_ids.contains(&id) { + continue; + } + if content_pick_ids.len() >= remaining { + break; + } + needed_document_ids.insert(id); + content_pick_ids.push(id); + } + + let documents: HashMap)> = + message_search_service::list_documents_by_conversation( + conn, + &needed_document_ids.iter().copied().collect::>(), + ) + .await? + .into_iter() + .map(|(conversation_id, text, blocks)| (conversation_id, (text, blocks))) + .collect(); + + let mut results = Vec::with_capacity(limit as usize); + for mut result in title_results { + let title_matches = build_title_match_locations( + result.summary.title.as_deref().unwrap_or_default(), + &terms, + ); + result.matches.extend(title_matches); + if candidate_set.contains(&result.summary.id) { + let (text, blocks) = documents + .get(&result.summary.id) + .map(|(text, blocks)| (text.as_str(), blocks.as_slice())) + .unwrap_or_default(); + result.matches + .extend(build_content_match_locations(text, blocks, &terms)); + let (snippet_prefix, snippet_match, snippet_suffix) = build_snippet(text, &terms); + result.snippet_prefix = snippet_prefix; + result.snippet_match = snippet_match; + result.snippet_suffix = snippet_suffix; + } + results.push(result); + } + + if !content_pick_ids.is_empty() { + let summaries = conversation_service::list_summaries_by_ids(conn, &content_pick_ids).await?; + let summary_by_id: HashMap = summaries + .into_iter() + .map(|summary| (summary.id, summary)) + .collect(); + for conversation_id in content_pick_ids { + let Some(summary) = summary_by_id.get(&conversation_id) else { + continue; + }; + let (text, blocks) = documents + .get(&conversation_id) + .map(|(text, blocks)| (text.as_str(), blocks.as_slice())) + .unwrap_or_default(); + let (snippet_prefix, snippet_match, snippet_suffix) = build_snippet(text, &terms); + let matches = build_content_match_locations(text, blocks, &terms); + results.push(DbConversationSearchResult { + summary: summary.clone(), + snippet_prefix, + snippet_match, + snippet_suffix, + matches, + }); + } + } + + Ok(results) +} + +/// A visibility scope shared by every term query, so LIMIT is applied after +/// folder / agent / lifecycle filtering instead of over the whole corpus. +struct Scope { + clause: String, + params: Vec, +} + +impl Scope { + fn build( + folder_ids: Option>, + agent_type: Option, + ) -> Result { + let mut clause = String::from( + "c.deleted_at IS NULL AND c.kind != 'loop' AND c.parent_id IS NULL", + ); + let mut params: Vec = Vec::new(); + match folder_ids { + Some(ids) if !ids.is_empty() => { + let placeholders = vec!["?"; ids.len()].join(", "); + clause.push_str(&format!(" AND c.folder_id IN ({placeholders})")); + for id in ids { + params.push(id.into()); + } + } + _ => { + clause.push_str( + " AND c.folder_id IN (SELECT id FROM folder WHERE deleted_at IS NULL)", + ); + } + } + if let Some(agent_type) = agent_type { + let agent_str = serde_json::to_value(agent_type) + .ok() + .and_then(|value| value.as_str().map(str::to_owned)) + .unwrap_or_default(); + clause.push_str(" AND c.agent_type = ?"); + params.push(agent_str.into()); + } + Ok(Self { clause, params }) + } + + async fn count(&self, conn: &DatabaseConnection) -> Result { + let sql = format!("SELECT COUNT(*) FROM conversation c WHERE {}", self.clause); + let row = conn + .query_one(Statement::from_sql_and_values( + DbBackend::Sqlite, + sql, + self.params.clone(), + )) + .await + .map_err(|err| AppCommandError::from(DbError::Database(err)))?; + let Some(row) = row else { + return Ok(0); + }; + row.try_get_by_index::(0) + .map_err(|err| AppCommandError::from(DbError::Database(err))) + } +} + +async fn term_candidates( + conn: &DatabaseConnection, + term: &str, + mode: &str, + short_fts_enabled: bool, + scope: &Scope, + cap: i64, +) -> Result)>, AppCommandError> { + if cap <= 0 { + return Ok(Vec::new()); + } + let char_len = term.chars().count(); + if mode == MODE_FTS && char_len >= 3 { + let Some(expression) = query::trigram_expression(term) else { + return scan_term_candidates(conn, term, scope, cap).await; + }; + return query_fts_candidates( + conn, + "message_search_trigram", + &expression, + term, + scope, + cap, + ) + .await; + } + + if mode == MODE_FTS && short_fts_enabled { + let expression = match query::short_query(term) { + ShortTermQuery::CjkUnigram { token } => format!("words : \"{token}\""), + ShortTermQuery::CjkBigram { phrase } => format!("bigrams : \"{phrase}\""), + ShortTermQuery::LatinPrefix { token } => format!("words : \"{token}\"*"), + }; + return query_fts_candidates(conn, "message_search_short", &expression, term, scope, cap) + .await; + } + + scan_term_candidates(conn, term, scope, cap).await +} + +async fn scan_term_candidates( + conn: &DatabaseConnection, + term: &str, + scope: &Scope, + cap: i64, +) -> Result)>, AppCommandError> { + let sql = format!( + "SELECT d.conversation_id, c.updated_at \ + FROM message_search_document d \ + JOIN conversation c ON c.id = d.conversation_id \ + WHERE instr(lower(d.text), lower(?)) > 0 \ + AND d.text LIKE ? ESCAPE '\\' \ + AND {} \ + ORDER BY c.updated_at DESC, d.conversation_id DESC \ + LIMIT ?", + scope.clause + ); + let mut params = vec![term.to_string().into(), query::like_pattern(term).into()]; + params.extend(scope.params.clone()); + params.push(cap.into()); + query_candidate_rows(conn, sql, params).await +} + +async fn query_fts_candidates( + conn: &DatabaseConnection, + table: &str, + expression: &str, + term: &str, + scope: &Scope, + cap: i64, +) -> Result)>, AppCommandError> { + // SQLite's LIKE only folds ASCII case, but the trigram tokenizer folds + // Unicode. Keep the cheap SQL exact filter for ASCII terms and verify + // non-ASCII terms in Rust so `ÉCOLE` stays a valid hit for `éco`. + let ascii = term.is_ascii(); + let select = if ascii { + "d.conversation_id, c.updated_at".to_string() + } else { + "d.conversation_id, c.updated_at, d.text".to_string() + }; + let exact = if ascii { + " AND d.text LIKE ? ESCAPE '\\'" + } else { + "" + }; + let sql = format!( + "SELECT {select} \ + FROM {table} \ + JOIN message_search_document d ON d.id = {table}.rowid \ + JOIN conversation c ON c.id = d.conversation_id \ + WHERE {table} MATCH ?{exact} AND {scope} \ + ORDER BY c.updated_at DESC, d.conversation_id DESC \ + LIMIT ?", + scope = scope.clause + ); + let mut params = vec![expression.to_string().into()]; + if ascii { + params.push(query::like_pattern(term).into()); + } + params.extend(scope.params.clone()); + params.push(cap.into()); + if ascii { + return query_candidate_rows(conn, sql, params).await; + } + let rows = query_candidate_rows_with_text(conn, sql, params).await?; + let lower = term.to_lowercase(); + Ok(rows + .into_iter() + .filter(|(_, _, text)| text.to_lowercase().contains(&lower)) + .map(|(id, updated_at, _)| (id, updated_at)) + .collect()) +} + +async fn query_candidate_rows( + conn: &DatabaseConnection, + sql: String, + params: Vec, +) -> Result)>, AppCommandError> { + let rows = conn + .query_all(Statement::from_sql_and_values(DbBackend::Sqlite, sql, params)) + .await + .map_err(|err| AppCommandError::from(DbError::Database(err)))?; + rows.into_iter() + .map(|row| { + Ok(( + row.try_get_by_index::(0)?, + row.try_get_by_index::>(1)?, + )) + }) + .collect::, sea_orm::DbErr>>() + .map_err(|err| AppCommandError::from(DbError::Database(err))) +} + +async fn query_candidate_rows_with_text( + conn: &DatabaseConnection, + sql: String, + params: Vec, +) -> Result, String)>, AppCommandError> { + let rows = conn + .query_all(Statement::from_sql_and_values(DbBackend::Sqlite, sql, params)) + .await + .map_err(|err| AppCommandError::from(DbError::Database(err)))?; + rows.into_iter() + .map(|row| { + Ok(( + row.try_get_by_index::(0)?, + row.try_get_by_index::>(1)?, + row.try_get_by_index::(2)?, + )) + }) + .collect::, sea_orm::DbErr>>() + .map_err(|err| AppCommandError::from(DbError::Database(err))) +} + +fn find_case_insensitive_char(haystack: &str, needle: &str) -> Option { + let hay: Vec = haystack.chars().collect(); + let needle: Vec = needle.chars().collect(); + if needle.is_empty() || needle.len() > hay.len() { + return None; + } + (0..=hay.len() - needle.len()).find(|start| { + hay[*start..*start + needle.len()] + .iter() + .zip(&needle) + .all(|(a, b)| a.to_lowercase().eq(b.to_lowercase())) + }) +} + +fn build_snippet(text: &str, terms: &[String]) -> (Option, Option, Option) { + let hay: Vec = text.chars().collect(); + let mut best: Option<(usize, usize)> = None; + for term in terms { + let needle: Vec = term.chars().collect(); + if let Some(start) = find_case_insensitive_char(text, term) { + if best.is_none_or(|(best_start, _)| start < best_start) { + best = Some((start, needle.len())); + } + } + } + let Some((start, len)) = best else { + return (None, None, None); + }; + let prefix_start = start.saturating_sub(SNIPPET_CONTEXT_CHARS); + let suffix_end = (start + len + SNIPPET_CONTEXT_CHARS).min(hay.len()); + let prefix = hay[prefix_start..start].iter().collect::(); + let matched = hay[start..start + len].iter().collect::(); + let suffix = hay[start + len..suffix_end].iter().collect::(); + ( + (!prefix.is_empty()).then_some(prefix), + Some(matched), + (!suffix.is_empty()).then_some(suffix), + ) +} + +fn build_title_match_locations(title: &str, terms: &[String]) -> Vec { + find_match_ranges(title, terms, MAX_MATCH_LOCATIONS) + .into_iter() + .map(|(char_start, char_end)| SearchMatchLocation { + kind: SearchMatchLocationKind::Title, + turn_id: None, + block_index: None, + char_start, + char_end, + }) + .collect() +} + +/// Title-only results for queries with no content hits (or content search +/// disabled): the same shape as a full result, with no snippet fields. +fn title_only_results( + summaries: Vec, + terms: &[String], + limit: usize, +) -> Vec { + summaries + .into_iter() + .take(limit) + .map(|summary| { + let matches = build_title_match_locations( + summary.title.as_deref().unwrap_or_default(), + terms, + ); + DbConversationSearchResult { + summary, + snippet_prefix: None, + snippet_match: None, + snippet_suffix: None, + matches, + } + }) + .collect() +} + +fn build_content_match_locations( + text: &str, + blocks: &[NormalizedBlockOffset], + terms: &[String], +) -> Vec { + find_match_ranges(text, terms, MAX_MATCH_LOCATIONS) + .into_iter() + .filter_map(|(char_start, char_end)| { + let block = blocks + .iter() + .find(|block| char_start >= block.start && char_end <= block.end)?; + Some(SearchMatchLocation { + kind: SearchMatchLocationKind::Content, + turn_id: Some(block.turn_id.clone()), + block_index: Some(block.block_index), + char_start: char_start - block.start + block.leading_trim, + char_end: char_end - block.start + block.leading_trim, + }) + }) + .collect() +} + +fn find_match_ranges(text: &str, terms: &[String], max_matches: usize) -> Vec<(usize, usize)> { + let hay: Vec = text.chars().collect(); + let mut ranges = Vec::new(); + + for term in terms { + let needle: Vec = term.chars().collect(); + if needle.is_empty() || needle.len() > hay.len() { + continue; + } + let mut start = 0; + while start + needle.len() <= hay.len() { + let matched = hay[start..start + needle.len()] + .iter() + .zip(&needle) + .all(|(left, right)| left.to_lowercase().eq(right.to_lowercase())); + if matched { + ranges.push((start, start + needle.len())); + start += needle.len(); + } else { + start += 1; + } + } + } + + ranges.sort_by_key(|(start, end)| (*start, *end)); + let mut deduped: Vec<(usize, usize)> = Vec::with_capacity(ranges.len()); + let mut last_end = None; + for (start, end) in ranges { + if last_end.is_some_and(|last| start < last) { + continue; + } + deduped.push((start, end)); + last_end = Some(end); + if deduped.len() >= max_matches { + break; + } + } + deduped +} + +pub async fn get_search_index_status_core( + conn: &DatabaseConnection, +) -> Result { + let state = message_search_service::ensure_search_state(conn).await?; + let visible = message_search_service::indexable_conversation_count(conn).await?; + let indexed_count = message_search_service::get_search_state(conn) + .await? + .indexed_conversation_count; + let progress = if !state.user_enabled || visible <= 0 { + 1.0 + } else { + (indexed_count as f64 / visible as f64).clamp(0.0, 1.0) + }; + Ok(SearchIndexStatus { + mode: state.mode, + user_enabled: state.user_enabled, + user_mode: state.user_mode, + indexed_conversation_count: indexed_count, + visible_conversation_count: visible, + building: false, + progress, + }) +} + +pub async fn set_search_settings_core( + conn: &DatabaseConnection, + enabled: bool, + user_mode: String, +) -> Result<(), AppCommandError> { + // Persist only: rebuilding / clearing the index is deferred to the + // background worker so the settings request never blocks on a full rebuild. + message_search_service::set_search_user_settings(conn, enabled, &user_mode).await?; + Ok(()) +} + +#[cfg(feature = "tauri-runtime")] +#[cfg_attr(feature = "tauri-runtime", tauri::command)] +pub async fn search_conversations( + db: tauri::State<'_, crate::db::AppDatabase>, + folder_ids: Option>, + agent_type: Option, + query: String, + limit: Option, +) -> Result, AppCommandError> { + search_conversations_core(&db.conn, folder_ids, agent_type, query, limit).await +} + +#[cfg(feature = "tauri-runtime")] +#[cfg_attr(feature = "tauri-runtime", tauri::command)] +pub async fn get_search_index_status( + db: tauri::State<'_, crate::db::AppDatabase>, +) -> Result { + get_search_index_status_core(&db.conn).await +} + +#[cfg(feature = "tauri-runtime")] +#[cfg_attr(feature = "tauri-runtime", tauri::command)] +pub async fn set_search_settings( + app: tauri::AppHandle, + db: tauri::State<'_, crate::db::AppDatabase>, + enabled: bool, + user_mode: String, +) -> Result<(), AppCommandError> { + set_search_settings_core(&db.conn, enabled, user_mode).await?; + { + use tauri::Manager; + if let Some(indexer) = + app.try_state::>() + { + indexer.request_mode_sync(); + } + } + Ok(()) +} +#[cfg(test)] +mod tests { + use super::*; + use crate::db::service::message_search_service::SyncFlags; + use crate::db::test_helpers::{fresh_in_memory_db, seed_conversation, seed_folder}; + use crate::search::normalizer::{NormalizedBlockOffset, NormalizedDocument}; + + fn doc(text: &str) -> NormalizedDocument { + NormalizedDocument { + text: text.to_string(), + content_hash: crate::search::normalizer::sha256_hex(text), + blocks: Vec::new(), + } + } + + fn doc_with_block(text: &str, turn_id: &str, block_index: usize) -> NormalizedDocument { + NormalizedDocument { + blocks: vec![NormalizedBlockOffset { + turn_id: turn_id.to_string(), + block_index, + start: 0, + end: text.chars().count(), + leading_trim: 0, + }], + text: text.to_string(), + content_hash: crate::search::normalizer::sha256_hex(text), + } + } + + #[tokio::test] + async fn scan_search_returns_content_snippet() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-command").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + message_search_service::upsert_document( + &db.conn, + conversation_id, + &doc_with_block("前缀 你好世界 后缀", "turn-1", 0), + None, + 1, + SyncFlags::default(), + ) + .await + .expect("doc"); + + let results = search_conversations_core( + &db.conn, + Some(vec![folder_id]), + None, + "世界".to_string(), + Some(10), + ) + .await + .expect("search"); + let result = results + .iter() + .find(|result| result.summary.id == conversation_id) + .expect("content result"); + assert_eq!(result.snippet_match.as_deref(), Some("世界")); + assert!(result.matches.iter().any(|match_location| { + match_location.kind == SearchMatchLocationKind::Content + && match_location.turn_id.as_deref() == Some("turn-1") + && match_location.block_index == Some(0) + && match_location.char_start == 5 + && match_location.char_end == 7 + })); + } + + #[tokio::test] + async fn title_matches_rank_before_content() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-title").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + conversation_service::update_title(&db.conn, conversation_id, "世界标题".to_string()) + .await + .expect("title"); + message_search_service::upsert_document( + &db.conn, + conversation_id, + &doc_with_block("正文 世界", "turn-1", 0), + None, + 1, + SyncFlags::default(), + ) + .await + .expect("doc"); + + let results = search_conversations_core( + &db.conn, + Some(vec![folder_id]), + None, + "世界".to_string(), + Some(10), + ) + .await + .expect("search"); + assert_eq!(results[0].summary.id, conversation_id); + assert_eq!(results[0].snippet_match.as_deref(), Some("世界")); + assert!( + results[0] + .matches + .iter() + .any(|match_location| match_location.kind == SearchMatchLocationKind::Title) + ); + assert!( + results[0] + .matches + .iter() + .any(|match_location| match_location.kind == SearchMatchLocationKind::Content) + ); + } + + #[tokio::test] + async fn fts_mode_uses_trigram_index() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-fts").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + message_search_service::set_search_mode(&db.conn, MODE_FTS, false) + .await + .expect("mode"); + message_search_service::upsert_document( + &db.conn, + conversation_id, + &doc("会话记录"), + None, + 1, + SyncFlags { + trigram: true, + short: false, + }, + ) + .await + .expect("doc"); + + let results = search_conversations_core( + &db.conn, + Some(vec![folder_id]), + None, + "会话".to_string(), + Some(10), + ) + .await + .expect("search"); + assert!( + results + .iter() + .any(|result| result.summary.id == conversation_id) + ); + } + + #[tokio::test] + async fn search_limit_applies_to_title_and_content_together() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-limit").await; + for index in 0..5 { + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + conversation_service::update_title( + &db.conn, + conversation_id, + format!("common {index}"), + ) + .await + .expect("title"); + } + + let results = search_conversations_core( + &db.conn, + Some(vec![folder_id]), + None, + "common".to_string(), + Some(2), + ) + .await + .expect("search"); + assert_eq!(results.len(), 2); + } + #[tokio::test] + async fn fts_trigram_path_finds_three_char_hits() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-trigram").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn).await.expect("state"); + message_search_service::set_search_mode(&db.conn, MODE_FTS, false).await.expect("mode"); + message_search_service::upsert_document( + &db.conn, conversation_id, &doc("会话记录全文"), None, 1, + SyncFlags { trigram: true, short: false }, + ) + .await + .expect("doc"); + let results = search_conversations_core( + &db.conn, Some(vec![folder_id]), None, "会话记录".to_string(), Some(10), + ) + .await + .expect("search"); + assert!(results.iter().any(|r| r.summary.id == conversation_id)); + } + + #[tokio::test] + async fn fts_short_table_supports_cjk_bigram_queries() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-short").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn).await.expect("state"); + message_search_service::set_search_mode(&db.conn, MODE_FTS, true).await.expect("mode"); + message_search_service::upsert_document( + &db.conn, conversation_id, &doc("搜索聊天记录"), None, 1, + SyncFlags { trigram: true, short: true }, + ) + .await + .expect("doc"); + let results = search_conversations_core( + &db.conn, Some(vec![folder_id]), None, "聊天".to_string(), Some(10), + ) + .await + .expect("search"); + assert!(results.iter().any(|r| r.summary.id == conversation_id)); + } + + #[tokio::test] + async fn folder_scope_filters_content_candidates() { + let db = fresh_in_memory_db().await; + let first = seed_folder(&db, "/tmp/search-folder-a").await; + let second = seed_folder(&db, "/tmp/search-folder-b").await; + let in_scope = seed_conversation(&db, first, AgentType::Codex).await; + let out_of_scope = seed_conversation(&db, second, AgentType::Codex).await; + for (conversation_id, text) in [(in_scope, "needle alpha"), (out_of_scope, "needle beta")] { + message_search_service::upsert_document( + &db.conn, conversation_id, &doc(text), None, 1, SyncFlags::default(), + ) + .await + .expect("doc"); + } + let results = search_conversations_core( + &db.conn, Some(vec![first]), None, "needle".to_string(), Some(10), + ) + .await + .expect("search"); + assert!(results.iter().any(|r| r.summary.id == in_scope)); + assert!(!results.iter().any(|r| r.summary.id == out_of_scope)); + } + + #[tokio::test] + async fn content_results_are_ordered_by_recent_activity() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-order").await; + let older = seed_conversation(&db, folder_id, AgentType::Codex).await; + let newer = seed_conversation(&db, folder_id, AgentType::Codex).await; + for (conversation_id, at) in [ + (older, chrono::Utc::now() - chrono::Duration::hours(2)), + (newer, chrono::Utc::now()), + ] { + conversation_service::refresh_external_activity(&db.conn, conversation_id, at, 1) + .await + .expect("activity"); + message_search_service::upsert_document( + &db.conn, conversation_id, &doc("common term"), None, 1, SyncFlags::default(), + ) + .await + .expect("doc"); + } + let results = search_conversations_core( + &db.conn, Some(vec![folder_id]), None, "common".to_string(), Some(10), + ) + .await + .expect("search"); + assert_eq!(results[0].summary.id, newer); + assert_eq!(results[1].summary.id, older); + } + + #[tokio::test] + async fn fts_folds_non_ascii_case_for_trigram_terms() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-accent").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn).await.expect("state"); + message_search_service::set_search_mode(&db.conn, MODE_FTS, true).await.expect("mode"); + message_search_service::upsert_document( + &db.conn, conversation_id, &doc("ÉCOLE 资料"), None, 1, + SyncFlags { trigram: true, short: true }, + ) + .await + .expect("doc"); + let results = search_conversations_core( + &db.conn, Some(vec![folder_id]), None, "éco".to_string(), Some(10), + ) + .await + .expect("search"); + assert!(results.iter().any(|r| r.summary.id == conversation_id)); + } +} \ No newline at end of file diff --git a/src-tauri/src/db/entities/message_search_document.rs b/src-tauri/src/db/entities/message_search_document.rs new file mode 100644 index 000000000..d5960d48d --- /dev/null +++ b/src-tauri/src/db/entities/message_search_document.rs @@ -0,0 +1,26 @@ +use sea_orm::entity::prelude::*; +use serde::{Deserialize, Serialize}; + +/// One normalized, searchable document per visible conversation. +/// +/// Only user and assistant text blocks are stored here; system prompts, +/// thinking, tool calls, and images are excluded by the normalizer before this +/// row is written. +#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Serialize, Deserialize)] +#[sea_orm(table_name = "message_search_document")] +pub struct Model { + #[sea_orm(primary_key)] + pub id: i32, + pub conversation_id: i32, + pub text: String, + pub content_hash: String, + pub block_offsets: String, + pub source_ended_at: Option, + pub source_message_count: i32, + pub updated_at: DateTimeUtc, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/src-tauri/src/db/entities/mod.rs b/src-tauri/src/db/entities/mod.rs index 1506c19c3..8dbe534b5 100644 --- a/src-tauri/src/db/entities/mod.rs +++ b/src-tauri/src/db/entities/mod.rs @@ -11,11 +11,13 @@ pub mod custom_agent; pub mod folder; pub mod folder_command; pub mod folder_link; +pub mod message_search_document; pub mod model_provider; pub mod opened_tab; pub mod prelude; pub mod quick_message; pub mod remote_workspace_connection; +pub mod search_index_state; pub mod token_usage_sync; pub mod token_usage_turn; pub mod work_task; diff --git a/src-tauri/src/db/entities/search_index_state.rs b/src-tauri/src/db/entities/search_index_state.rs new file mode 100644 index 000000000..e036d2480 --- /dev/null +++ b/src-tauri/src/db/entities/search_index_state.rs @@ -0,0 +1,23 @@ +use sea_orm::entity::prelude::*; +use serde::{Deserialize, Serialize}; + +/// Singleton row describing the content-search index lifecycle. +#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Serialize, Deserialize)] +#[sea_orm(table_name = "search_index_state")] +pub struct Model { + #[sea_orm(primary_key)] + pub id: i32, + pub schema_version: i32, + pub mode: String, + pub threshold_mb: f64, + pub short_fts_enabled: bool, + pub indexed_conversation_count: i32, + pub last_backfill_at: Option, + pub user_enabled: bool, + pub user_mode: String, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/src-tauri/src/db/migration/m20260814_000001_message_search.rs b/src-tauri/src/db/migration/m20260814_000001_message_search.rs new file mode 100644 index 000000000..5d72be444 --- /dev/null +++ b/src-tauri/src/db/migration/m20260814_000001_message_search.rs @@ -0,0 +1,210 @@ +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager + .create_table( + Table::create() + .table(MessageSearchDocument::Table) + .if_not_exists() + .col( + ColumnDef::new(MessageSearchDocument::Id) + .integer() + .not_null() + .auto_increment() + .primary_key(), + ) + .col( + ColumnDef::new(MessageSearchDocument::ConversationId) + .integer() + .not_null(), + ) + .col( + ColumnDef::new(MessageSearchDocument::Text) + .text() + .not_null(), + ) + .col( + ColumnDef::new(MessageSearchDocument::ContentHash) + .string() + .not_null(), + ) + .col( + ColumnDef::new(MessageSearchDocument::SourceEndedAt) + .timestamp_with_time_zone() + .null(), + ) + .col( + ColumnDef::new(MessageSearchDocument::SourceMessageCount) + .integer() + .not_null() + .default(0), + ) + .col( + ColumnDef::new(MessageSearchDocument::UpdatedAt) + .timestamp_with_time_zone() + .not_null(), + ) + .to_owned(), + ) + .await?; + + manager + .create_index( + Index::create() + .if_not_exists() + .name("idx_message_search_document_conversation") + .table(MessageSearchDocument::Table) + .col(MessageSearchDocument::ConversationId) + .unique() + .to_owned(), + ) + .await?; + + manager + .create_table( + Table::create() + .table(SearchIndexState::Table) + .if_not_exists() + .col( + ColumnDef::new(SearchIndexState::Id) + .integer() + .not_null() + .primary_key(), + ) + .col( + ColumnDef::new(SearchIndexState::SchemaVersion) + .integer() + .not_null() + .default(1), + ) + .col( + ColumnDef::new(SearchIndexState::Mode) + .string() + .not_null() + .default("scan"), + ) + .col( + ColumnDef::new(SearchIndexState::ThresholdMb) + .double() + .not_null() + .default(40.0), + ) + .col( + ColumnDef::new(SearchIndexState::ShortFtsEnabled) + .boolean() + .not_null() + .default(false), + ) + .col( + ColumnDef::new(SearchIndexState::ShortThresholdMb) + .double() + .not_null() + .default(40.0), + ) + .col(ColumnDef::new(SearchIndexState::ScanMsPerMb).double().null()) + .col( + ColumnDef::new(SearchIndexState::IndexedConversationCount) + .integer() + .not_null() + .default(0), + ) + .col( + ColumnDef::new(SearchIndexState::LastCalibrationAt) + .timestamp_with_time_zone() + .null(), + ) + .col( + ColumnDef::new(SearchIndexState::LastBackfillAt) + .timestamp_with_time_zone() + .null(), + ) + .col( + ColumnDef::new(SearchIndexState::UserEnabled) + .boolean() + .not_null() + .default(true), + ) + .col( + ColumnDef::new(SearchIndexState::UserMode) + .string() + .not_null() + .default("auto"), + ) + .to_owned(), + ) + .await?; + + let conn = manager.get_connection(); + conn.execute_unprepared( + "CREATE VIRTUAL TABLE IF NOT EXISTS message_search_trigram USING fts5(\ + text, content='', contentless_delete=1, detail=none, tokenize='trigram')", + ) + .await?; + conn.execute_unprepared( + "CREATE VIRTUAL TABLE IF NOT EXISTS message_search_short USING fts5(\ + words, bigrams, content='', contentless_delete=1, detail=none, \ + tokenize='unicode61 remove_diacritics 2')", + ) + .await?; + + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let conn = manager.get_connection(); + conn.execute_unprepared("DROP TABLE IF EXISTS message_search_trigram") + .await?; + conn.execute_unprepared("DROP TABLE IF EXISTS message_search_short") + .await?; + manager + .drop_table( + Table::drop() + .table(MessageSearchDocument::Table) + .if_exists() + .to_owned(), + ) + .await?; + manager + .drop_table( + Table::drop() + .table(SearchIndexState::Table) + .if_exists() + .to_owned(), + ) + .await + } +} + +#[derive(DeriveIden)] +enum MessageSearchDocument { + Table, + Id, + ConversationId, + Text, + ContentHash, + SourceEndedAt, + SourceMessageCount, + UpdatedAt, +} + +#[derive(DeriveIden)] +enum SearchIndexState { + Table, + Id, + SchemaVersion, + Mode, + ThresholdMb, + ShortFtsEnabled, + ShortThresholdMb, + ScanMsPerMb, + IndexedConversationCount, + LastCalibrationAt, + LastBackfillAt, + UserEnabled, + UserMode, +} diff --git a/src-tauri/src/db/migration/m20260815_000001_search_match_locations.rs b/src-tauri/src/db/migration/m20260815_000001_search_match_locations.rs new file mode 100644 index 000000000..0fb1561b2 --- /dev/null +++ b/src-tauri/src/db/migration/m20260815_000001_search_match_locations.rs @@ -0,0 +1,66 @@ +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager + .alter_table( + Table::alter() + .table(MessageSearchDocument::Table) + .add_column( + ColumnDef::new(MessageSearchDocument::BlockOffsets) + .text() + .not_null() + .default("[]"), + ) + .to_owned(), + ) + .await?; + + // The new location metadata changes the normalized document format. + // Bump the singleton search schema version so the background indexer + // rebuilds old documents and fills the new column with real offsets. + manager + .get_connection() + .execute_unprepared("DELETE FROM message_search_trigram") + .await?; + manager + .get_connection() + .execute_unprepared("DELETE FROM message_search_short") + .await?; + manager + .get_connection() + .execute_unprepared("DELETE FROM message_search_document") + .await?; + manager + .get_connection() + .execute_unprepared("UPDATE search_index_state SET schema_version = 2 WHERE id = 1") + .await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager + .alter_table( + Table::alter() + .table(MessageSearchDocument::Table) + .drop_column(MessageSearchDocument::BlockOffsets) + .to_owned(), + ) + .await?; + manager + .get_connection() + .execute_unprepared("UPDATE search_index_state SET schema_version = 1 WHERE id = 1") + .await?; + Ok(()) + } +} + +#[derive(DeriveIden)] +enum MessageSearchDocument { + Table, + BlockOffsets, +} diff --git a/src-tauri/src/db/migration/m20260818_000001_short_fts_detail_column.rs b/src-tauri/src/db/migration/m20260818_000001_short_fts_detail_column.rs new file mode 100644 index 000000000..2ba8ce38b --- /dev/null +++ b/src-tauri/src/db/migration/m20260818_000001_short_fts_detail_column.rs @@ -0,0 +1,110 @@ +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let conn = manager.get_connection(); + + // `detail=none` rejects column-filtered MATCH queries (`words :`, `bigrams :`), + // which the short-query path relies on. Recreate the table with + // `detail=column` so those queries work. Schema upgrade + repopulation are + // handled by the indexer once it observes the older schema_version below. + conn.execute_unprepared("DROP TABLE IF EXISTS message_search_short") + .await?; + conn.execute_unprepared( + "CREATE VIRTUAL TABLE message_search_short USING fts5(\ + words, bigrams, content='', contentless_delete=1, detail=column, \ + tokenize='unicode61 remove_diacritics 2')", + ) + .await?; + + // Columns reserved for the calibration / p95 watchdog / independent short + // threshold design were never read or written. Drop them instead of + // shipping dead schema. + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .drop_column(SearchIndexState::ScanMsPerMb) + .to_owned(), + ) + .await?; + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .drop_column(SearchIndexState::LastCalibrationAt) + .to_owned(), + ) + .await?; + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .drop_column(SearchIndexState::ShortThresholdMb) + .to_owned(), + ) + .await?; + + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let conn = manager.get_connection(); + conn.execute_unprepared("DROP TABLE IF EXISTS message_search_short") + .await?; + conn.execute_unprepared( + "CREATE VIRTUAL TABLE message_search_short USING fts5(\ + words, bigrams, content='', contentless_delete=1, detail=none, \ + tokenize='unicode61 remove_diacritics 2')", + ) + .await?; + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .add_column( + ColumnDef::new(SearchIndexState::ScanMsPerMb).double().null(), + ) + .to_owned(), + ) + .await?; + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .add_column( + ColumnDef::new(SearchIndexState::LastCalibrationAt) + .timestamp_with_time_zone() + .null(), + ) + .to_owned(), + ) + .await?; + manager + .alter_table( + Table::alter() + .table(SearchIndexState::Table) + .add_column( + ColumnDef::new(SearchIndexState::ShortThresholdMb) + .double() + .not_null() + .default(40.0), + ) + .to_owned(), + ) + .await?; + Ok(()) + } +} + +#[derive(DeriveIden)] +enum SearchIndexState { + Table, + ScanMsPerMb, + LastCalibrationAt, + ShortThresholdMb, +} diff --git a/src-tauri/src/db/migration/mod.rs b/src-tauri/src/db/migration/mod.rs index 9e4989b9b..fb0572d42 100644 --- a/src-tauri/src/db/migration/mod.rs +++ b/src-tauri/src/db/migration/mod.rs @@ -37,7 +37,10 @@ mod m20260803_000001_folder_link; mod m20260803_000001_token_usage; mod m20260807_000001_work_task_scheduled_at; mod m20260808_000001_custom_agent_supports_mcp; +mod m20260814_000001_message_search; +mod m20260815_000001_search_match_locations; mod m20260817_000001_work_task_conversation_title; +mod m20260818_000001_short_fts_detail_column; pub struct Migrator; #[async_trait::async_trait] @@ -81,7 +84,10 @@ impl MigratorTrait for Migrator { Box::new(m20260803_000001_token_usage::Migration), Box::new(m20260807_000001_work_task_scheduled_at::Migration), Box::new(m20260808_000001_custom_agent_supports_mcp::Migration), + Box::new(m20260814_000001_message_search::Migration), + Box::new(m20260815_000001_search_match_locations::Migration), Box::new(m20260817_000001_work_task_conversation_title::Migration), + Box::new(m20260818_000001_short_fts_detail_column::Migration), ] } } diff --git a/src-tauri/src/db/service/conversation_service.rs b/src-tauri/src/db/service/conversation_service.rs index dd9e85fa8..511e3726c 100644 --- a/src-tauri/src/db/service/conversation_service.rs +++ b/src-tauri/src/db/service/conversation_service.rs @@ -952,6 +952,26 @@ pub async fn get_by_delegation_call_id( Ok(conv.map(conv_to_summary)) } +/// Fetch full summaries for a bounded set of ids in one query (used by content +/// search to build only the rows it will actually return, instead of pulling +/// every workspace summary). Preserves caller-side ordering responsibility: +/// rows come back unsorted and callers reorder by their own ranking. +pub async fn list_summaries_by_ids( + conn: &DatabaseConnection, + ids: &[i32], +) -> Result, DbError> { + if ids.is_empty() { + return Ok(Vec::new()); + } + let rows = conversation::Entity::find() + .filter(conversation::Column::DeletedAt.is_null()) + .filter(conversation::Column::Id.is_in(ids.to_vec())) + .all(conn) + .await?; + let mut summaries: Vec = rows.into_iter().map(conv_to_summary).collect(); + fill_child_counts(conn, &mut summaries).await?; + Ok(summaries) +} pub async fn list_by_folder( conn: &DatabaseConnection, folder_id: i32, diff --git a/src-tauri/src/db/service/message_search_service.rs b/src-tauri/src/db/service/message_search_service.rs new file mode 100644 index 000000000..738a5ab6c --- /dev/null +++ b/src-tauri/src/db/service/message_search_service.rs @@ -0,0 +1,738 @@ +use chrono::Utc; +use sea_orm::{ + ActiveModelTrait, ActiveValue::NotSet, ColumnTrait, ConnectionTrait, DatabaseConnection, + DbBackend, EntityTrait, PaginatorTrait, QueryFilter, Set, Statement, + TransactionTrait, +}; + +use crate::db::entities::{ + conversation, conversation::ConversationKind, folder, message_search_document, + search_index_state, +}; +use crate::db::error::DbError; +use crate::models::AgentType; +use crate::search::normalizer::{NormalizedBlockOffset, NormalizedDocument}; + +pub const SEARCH_SCHEMA_VERSION: i32 = 3; +pub const MODE_SCAN: &str = "scan"; +pub const MODE_FTS: &str = "fts"; +pub const USER_MODE_AUTO: &str = "auto"; +pub const USER_MODE_SCAN: &str = "scan"; +pub const USER_MODE_FTS: &str = "fts"; + +/// Which optional FTS tables should be kept in sync with a document write. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct SyncFlags { + pub trigram: bool, + pub short: bool, +} + +pub async fn ensure_search_state( + conn: &DatabaseConnection, +) -> Result { + if let Some(state) = search_index_state::Entity::find_by_id(1).one(conn).await? { + return Ok(state); + } + let state = search_index_state::ActiveModel { + id: Set(1), + schema_version: Set(SEARCH_SCHEMA_VERSION), + mode: Set(MODE_SCAN.to_string()), + threshold_mb: Set(40.0), + short_fts_enabled: Set(false), + indexed_conversation_count: Set(0), + last_backfill_at: NotSet, + user_enabled: Set(true), + user_mode: Set(USER_MODE_AUTO.to_string()), + }; + Ok(state.insert(conn).await?) +} + +pub async fn get_search_state( + conn: &C, +) -> Result +where + C: ConnectionTrait, +{ + search_index_state::Entity::find_by_id(1) + .one(conn) + .await? + .ok_or_else(|| DbError::NotFound("search_index_state row is missing".to_string())) +} + +pub async fn set_search_mode( + conn: &DatabaseConnection, + mode: &str, + short_fts_enabled: bool, +) -> Result<(), DbError> { + let Some(state) = search_index_state::Entity::find_by_id(1).one(conn).await? else { + return Err(DbError::NotFound( + "search_index_state row is missing".to_string(), + )); + }; + if !matches!(mode, MODE_SCAN | MODE_FTS) { + return Err(DbError::Validation(format!("invalid search mode: {mode}"))); + } + let mut active: search_index_state::ActiveModel = state.into(); + active.mode = Set(mode.to_string()); + active.short_fts_enabled = Set(short_fts_enabled); + active.update(conn).await?; + Ok(()) +} + +/// Atomically rebuild the optional FTS postings and switch modes in one +/// transaction. If the rebuild fails, the rollback keeps the previous mode and +/// tables intact so the indexer can retry on its next tick instead of being +/// stranded in `fts` with empty postings. +pub async fn rebuild_fts_and_set_mode( + conn: &DatabaseConnection, + mode: &str, + short_fts_enabled: bool, +) -> Result<(), DbError> { + if !matches!(mode, MODE_SCAN | MODE_FTS) { + return Err(DbError::Validation(format!("invalid search mode: {mode}"))); + } + let txn = conn.begin().await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_trigram".to_string(), + )) + .await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_short".to_string(), + )) + .await?; + let sync = SyncFlags { + trigram: mode == MODE_FTS, + short: mode == MODE_FTS && short_fts_enabled, + }; + let documents = message_search_document::Entity::find().all(&txn).await?; + for model in &documents { + let doc = NormalizedDocument { + text: model.text.clone(), + content_hash: model.content_hash.clone(), + blocks: parse_block_offsets(&model.block_offsets), + }; + sync_fts_rows(&txn, model.id, &doc, sync).await?; + } + let state = get_search_state(&txn).await?; + let mut active: search_index_state::ActiveModel = state.into(); + active.mode = Set(mode.to_string()); + active.short_fts_enabled = Set(short_fts_enabled); + active.update(&txn).await?; + txn.commit().await?; + Ok(()) +} + +/// Bring a pre-v3 schema (or an otherwise stale singleton row) up to date and +/// repopulate the FTS postings that the migration had to drop. +pub async fn upgrade_schema_if_needed(conn: &DatabaseConnection) -> Result { + let state = ensure_search_state(conn).await?; + if state.schema_version >= SEARCH_SCHEMA_VERSION { + return Ok(false); + } + let sync = SyncFlags { + trigram: state.mode == MODE_FTS, + short: state.mode == MODE_FTS && state.short_fts_enabled, + }; + rebuild_fts_from_documents(conn, sync).await?; + let mut active: search_index_state::ActiveModel = state.into(); + active.schema_version = Set(SEARCH_SCHEMA_VERSION); + active.update(conn).await?; + Ok(true) +} + +/// Drop every stored content document and its FTS rows. Used when content +/// search is disabled so plaintext transcripts are not retained in the DB. +pub async fn clear_all_documents(conn: &DatabaseConnection) -> Result<(), DbError> { + let txn = conn.begin().await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_trigram".to_string(), + )) + .await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_short".to_string(), + )) + .await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_document".to_string(), + )) + .await?; + let state = get_search_state(&txn).await?; + let mut active: search_index_state::ActiveModel = state.into(); + active.indexed_conversation_count = Set(0); + active.update(&txn).await?; + txn.commit().await?; + Ok(()) +} + +/// Reclaim documents whose conversation no longer satisfies the visibility +/// filters (soft-deleted, loop, delegation child, or missing row). Deletions +/// performed while the app was not running have no event to consume, so this +/// sweep is the backstop. +pub async fn delete_orphan_documents(conn: &DatabaseConnection) -> Result { + let state = ensure_search_state(conn).await?; + let sync = SyncFlags { + trigram: state.mode == MODE_FTS, + short: state.mode == MODE_FTS && state.short_fts_enabled, + }; + let orphans: Vec = conn + .query_all(Statement::from_string( + DbBackend::Sqlite, + "SELECT d.id FROM message_search_document d \ + LEFT JOIN conversation c ON c.id = d.conversation_id \ + WHERE c.id IS NULL OR c.deleted_at IS NOT NULL \ + OR c.parent_id IS NOT NULL OR c.kind = 'loop'" + .to_string(), + )) + .await? + .into_iter() + .map(|row| row.try_get_by_index::(0)) + .collect::, _>>()?; + if orphans.is_empty() { + return Ok(0); + } + let txn = conn.begin().await?; + for id in &orphans { + delete_fts_rows(&txn, *id, sync).await?; + message_search_document::Entity::delete_by_id(*id) + .exec(&txn) + .await?; + } + txn.commit().await?; + Ok(orphans.len() as u64) +} + +pub async fn set_search_user_settings( + conn: &DatabaseConnection, + enabled: bool, + user_mode: &str, +) -> Result<(), DbError> { + let Some(state) = search_index_state::Entity::find_by_id(1).one(conn).await? else { + return Err(DbError::NotFound( + "search_index_state row is missing".to_string(), + )); + }; + if !matches!(user_mode, USER_MODE_AUTO | USER_MODE_SCAN | USER_MODE_FTS) { + return Err(DbError::Validation(format!( + "invalid user search mode: {user_mode}" + ))); + } + let mut active: search_index_state::ActiveModel = state.into(); + active.user_enabled = Set(enabled); + active.user_mode = Set(user_mode.to_string()); + active.update(conn).await?; + Ok(()) +} + +pub async fn set_index_progress( + conn: &DatabaseConnection, + indexed_conversation_count: i32, + backfill_at: Option>, +) -> Result<(), DbError> { + let Some(state) = search_index_state::Entity::find_by_id(1).one(conn).await? else { + return Err(DbError::NotFound( + "search_index_state row is missing".to_string(), + )); + }; + let mut active: search_index_state::ActiveModel = state.into(); + active.indexed_conversation_count = Set(indexed_conversation_count); + if let Some(backfill_at) = backfill_at { + active.last_backfill_at = Set(Some(backfill_at)); + } + active.update(conn).await?; + Ok(()) +} + +/// Upsert one conversation document and its enabled FTS rows atomically. +/// +/// Returns the stable document row id, which is also both FTS tables' rowid. +pub async fn upsert_document( + conn: &DatabaseConnection, + conversation_id: i32, + doc: &NormalizedDocument, + source_ended_at: Option>, + source_message_count: i32, + sync: SyncFlags, +) -> Result { + let txn = conn.begin().await?; + let existing = message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.eq(conversation_id)) + .one(&txn) + .await?; + + let id = if let Some(existing) = existing { + let id = existing.id; + let mut active: message_search_document::ActiveModel = existing.into(); + active.text = Set(doc.text.clone()); + active.content_hash = Set(doc.content_hash.clone()); + active.block_offsets = Set(block_offsets_json(doc)?); + active.source_ended_at = Set(source_ended_at); + active.source_message_count = Set(source_message_count); + active.updated_at = Set(Utc::now()); + active.update(&txn).await?; + id + } else { + let now = Utc::now(); + let model = message_search_document::ActiveModel { + id: NotSet, + conversation_id: Set(conversation_id), + text: Set(doc.text.clone()), + content_hash: Set(doc.content_hash.clone()), + block_offsets: Set(block_offsets_json(doc)?), + source_ended_at: Set(source_ended_at), + source_message_count: Set(source_message_count), + updated_at: Set(now), + }; + model.insert(&txn).await?.id + }; + + sync_fts_rows(&txn, id, doc, sync).await?; + txn.commit().await?; + Ok(id) +} + +fn block_offsets_json(doc: &NormalizedDocument) -> Result { + serde_json::to_string(&doc.blocks) + .map_err(|error| DbError::Migration(format!("serialize block offsets: {error}"))) +} + +pub async fn delete_document( + conn: &DatabaseConnection, + conversation_id: i32, + sync: SyncFlags, +) -> Result<(), DbError> { + let txn = conn.begin().await?; + if let Some(existing) = message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.eq(conversation_id)) + .one(&txn) + .await? + { + delete_fts_rows(&txn, existing.id, sync).await?; + message_search_document::Entity::delete_by_id(existing.id) + .exec(&txn) + .await?; + } + txn.commit().await?; + Ok(()) +} + +async fn sync_fts_rows( + conn: &C, + id: i32, + doc: &NormalizedDocument, + sync: SyncFlags, +) -> Result<(), DbError> +where + C: ConnectionTrait, +{ + delete_fts_rows(conn, id, sync).await?; + if sync.trigram { + conn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO message_search_trigram(rowid, text) VALUES(?, ?)", + [id.into(), doc.text.clone().into()], + )) + .await?; + } + if sync.short { + let tokens = crate::search::normalizer::short_index_tokens(&doc.text); + conn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO message_search_short(rowid, words, bigrams) VALUES(?, ?, ?)", + [ + id.into(), + tokens.words.clone().into(), + tokens.bigrams.clone().into(), + ], + )) + .await?; + } + Ok(()) +} + +async fn delete_fts_rows(conn: &C, id: i32, sync: SyncFlags) -> Result<(), DbError> +where + C: ConnectionTrait, +{ + if sync.trigram { + conn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "DELETE FROM message_search_trigram WHERE rowid = ?", + [id.into()], + )) + .await?; + } + if sync.short { + conn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "DELETE FROM message_search_short WHERE rowid = ?", + [id.into()], + )) + .await?; + } + Ok(()) +} + +pub async fn total_indexed_text_bytes(conn: &DatabaseConnection) -> Result { + let row = conn + .query_one(Statement::from_string( + DbBackend::Sqlite, + "SELECT COALESCE(SUM(LENGTH(CAST(text AS BLOB))), 0) \ + FROM message_search_document" + .to_string(), + )) + .await? + .map(|result| result.try_get_by_index::(0)) + .transpose()? + .unwrap_or(0); + Ok(row) +} + +/// Count conversations that are searchable under the same filters the +/// existing title search applies. +pub async fn visible_conversation_count( + conn: &DatabaseConnection, + folder_ids: Option>, + agent_type: Option, +) -> Result { + let mut query = conversation::Entity::find() + .filter(conversation::Column::DeletedAt.is_null()) + .filter(conversation::Column::Kind.ne(ConversationKind::Loop)) + .filter(conversation::Column::ParentId.is_null()); + + match folder_ids { + Some(ids) if !ids.is_empty() => { + query = query.filter(conversation::Column::FolderId.is_in(ids)); + } + Some(_) => return Ok(0), + None => { + let active_ids: Vec = folder::Entity::find() + .filter(folder::Column::DeletedAt.is_null()) + .all(conn) + .await? + .into_iter() + .map(|folder| folder.id) + .collect(); + if active_ids.is_empty() { + return Ok(0); + } + query = query.filter(conversation::Column::FolderId.is_in(active_ids)); + } + } + + if let Some(agent_type) = agent_type { + let agent_str = serde_json::to_value(agent_type) + .ok() + .and_then(|value| value.as_str().map(str::to_owned)) + .unwrap_or_default(); + query = query.filter(conversation::Column::AgentType.eq(agent_str)); + } + + Ok(i64::try_from(query.count(conn).await?).unwrap_or(i64::MAX)) +} + +pub async fn indexable_conversation_count(conn: &DatabaseConnection) -> Result { + Ok(i64::try_from( + conversation::Entity::find() + .filter(conversation::Column::DeletedAt.is_null()) + .filter(conversation::Column::Kind.ne(ConversationKind::Loop)) + .filter(conversation::Column::ParentId.is_null()) + .filter(conversation::Column::ExternalId.is_not_null()) + .count(conn) + .await?, + ) + .unwrap_or(i64::MAX)) +} + +pub async fn list_documents_by_conversation( + conn: &DatabaseConnection, + conversation_ids: &[i32], +) -> Result)>, DbError> { + if conversation_ids.is_empty() { + return Ok(Vec::new()); + } + Ok(message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.is_in(conversation_ids.to_vec())) + .all(conn) + .await? + .into_iter() + .map(|model| { + ( + model.conversation_id, + model.text, + parse_block_offsets(&model.block_offsets), + ) + }) + .collect()) +} + +/// Decode indexed block offsets. Legacy rows fall back to an empty manifest, +/// which keeps search working but disables precise jump/highlighting. +pub fn parse_block_offsets(raw: &str) -> Vec { + serde_json::from_str(raw).unwrap_or_default() +} + +/// Rebuild the optional FTS postings from the authoritative document table. +/// +/// Used when switching into FTS mode; rows for disabled tables are cleared. +pub async fn rebuild_fts_from_documents( + conn: &DatabaseConnection, + sync: SyncFlags, +) -> Result<(), DbError> { + let txn = conn.begin().await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_trigram".to_string(), + )) + .await?; + txn.execute(Statement::from_string( + DbBackend::Sqlite, + "DELETE FROM message_search_short".to_string(), + )) + .await?; + + if sync.trigram || sync.short { + let documents = message_search_document::Entity::find().all(&txn).await?; + for model in documents { + let doc = NormalizedDocument { + text: model.text.clone(), + content_hash: model.content_hash.clone(), + blocks: parse_block_offsets(&model.block_offsets), + }; + if sync.trigram { + txn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO message_search_trigram(rowid, text) VALUES(?, ?)", + [model.id.into(), doc.text.clone().into()], + )) + .await?; + } + if sync.short { + let tokens = crate::search::normalizer::short_index_tokens(&doc.text); + txn.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO message_search_short(rowid, words, bigrams) VALUES(?, ?, ?)", + [model.id.into(), tokens.words.into(), tokens.bigrams.into()], + )) + .await?; + } + } + } + txn.commit().await?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::db::test_helpers::{fresh_in_memory_db, seed_conversation, seed_folder}; + use crate::search::normalizer::{NormalizedBlockOffset, NormalizedDocument}; + + fn doc(text: &str) -> NormalizedDocument { + NormalizedDocument { + text: text.to_string(), + content_hash: crate::search::normalizer::sha256_hex(text), + blocks: Vec::new(), + } + } + + #[tokio::test] + async fn upsert_persists_block_offsets() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-offsets").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + let doc = NormalizedDocument { + text: "hello".to_string(), + content_hash: crate::search::normalizer::sha256_hex("hello"), + blocks: vec![NormalizedBlockOffset { + turn_id: "turn-1".to_string(), + block_index: 0, + start: 0, + end: 5, + leading_trim: 0, + }], + }; + + upsert_document( + &db.conn, + conversation_id, + &doc, + None, + 1, + SyncFlags::default(), + ) + .await + .expect("insert"); + + let rows = list_documents_by_conversation(&db.conn, &[conversation_id]) + .await + .expect("documents"); + assert_eq!(rows[0].2[0].turn_id, "turn-1"); + assert_eq!(rows[0].2[0].block_index, 0); + } + + #[tokio::test] + async fn state_starts_with_scan_defaults() { + let db = fresh_in_memory_db().await; + let state = ensure_search_state(&db.conn).await.expect("state"); + assert_eq!(state.id, 1); + assert_eq!(state.mode, MODE_SCAN); + assert_eq!(state.threshold_mb, 40.0); + assert!(state.user_enabled); + assert_eq!(state.user_mode, USER_MODE_AUTO); + } + + #[tokio::test] + async fn upsert_preserves_id_and_updates_hash() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-service").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + + let first_id = upsert_document( + &db.conn, + conversation_id, + &doc("first"), + None, + 1, + SyncFlags { + trigram: true, + short: true, + }, + ) + .await + .expect("insert"); + + let second_id = upsert_document( + &db.conn, + conversation_id, + &doc("second"), + None, + 2, + SyncFlags { + trigram: true, + short: true, + }, + ) + .await + .expect("update"); + + assert_eq!(first_id, second_id); + let rows = list_documents_by_conversation(&db.conn, &[conversation_id]) + .await + .expect("documents"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].1, "second"); + } + + #[tokio::test] + async fn delete_removes_fts_rows_and_document() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-delete").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::ClaudeCode).await; + let id = upsert_document( + &db.conn, + conversation_id, + &doc("delete me"), + None, + 1, + SyncFlags { + trigram: true, + short: true, + }, + ) + .await + .expect("insert"); + + delete_document( + &db.conn, + conversation_id, + SyncFlags { + trigram: true, + short: true, + }, + ) + .await + .expect("delete"); + + let orphan: i64 = db + .conn + .query_one(Statement::from_sql_and_values( + DbBackend::Sqlite, + "SELECT COUNT(*) FROM message_search_trigram WHERE rowid = ?", + [id.into()], + )) + .await + .expect("count") + .unwrap() + .try_get_by_index::(0) + .expect("count value"); + assert_eq!(orphan, 0); + } + + #[tokio::test] + async fn visible_count_filters_children_loops_and_deleted() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-count").await; + let first = seed_conversation(&db, folder_id, AgentType::Codex).await; + seed_conversation(&db, folder_id, AgentType::Codex).await; + crate::db::service::conversation_service::create_with_delegation( + &db.conn, + folder_id, + AgentType::Codex, + Some("child".into()), + None, + Some(crate::acp::delegation::spawner::DelegationLink { + parent_conversation_id: first, + parent_tool_use_id: "tu".into(), + delegation_call_id: "call".into(), + }), + ) + .await + .expect("child"); + + let count = visible_conversation_count(&db.conn, Some(vec![folder_id]), None) + .await + .expect("count"); + assert_eq!(count, 2); + } + #[tokio::test] + async fn upgrade_schema_marks_current_version() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-upgrade").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + let state = ensure_search_state(&db.conn).await.expect("state"); + let mut active: search_index_state::ActiveModel = state.into(); + active.schema_version = Set(2); + active.update(&db.conn).await.expect("downgrade version"); + + let upgraded = upgrade_schema_if_needed(&db.conn).await.expect("upgrade"); + assert!(upgraded); + let state = get_search_state(&db.conn).await.expect("state"); + assert_eq!(state.schema_version, SEARCH_SCHEMA_VERSION); + assert!(upgrade_schema_if_needed(&db.conn).await.expect("again") == false); + drop(conversation_id); + } + + #[tokio::test] + async fn orphan_documents_are_reclaimed() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-orphan").await; + let conversation_id = seed_conversation(&db, folder_id, AgentType::Codex).await; + ensure_search_state(&db.conn).await.expect("state"); + upsert_document(&db.conn, conversation_id, &doc("orphan text"), None, 1, SyncFlags::default()) + .await + .expect("doc"); + crate::db::service::conversation_service::soft_delete(&db.conn, conversation_id) + .await + .expect("delete"); + + let removed = delete_orphan_documents(&db.conn).await.expect("sweep"); + assert_eq!(removed, 1); + let count = message_search_document::Entity::find().count(&db.conn).await.expect("count"); + assert_eq!(count, 0); + } +} diff --git a/src-tauri/src/db/service/mod.rs b/src-tauri/src/db/service/mod.rs index edda1d933..b65828537 100644 --- a/src-tauri/src/db/service/mod.rs +++ b/src-tauri/src/db/service/mod.rs @@ -10,6 +10,7 @@ pub mod folder_link_service; pub mod folder_service; pub mod import_service; pub mod model_provider_service; +pub mod message_search_service; pub mod quick_message_service; pub mod remote_workspace_connection_service; pub mod sender_context_service; diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index f780ebbb0..a40f9e41c 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -37,6 +37,7 @@ pub mod pets; #[cfg(feature = "tauri-runtime")] pub mod preferences; pub mod process; +pub mod search; pub mod supervise; mod terminal; pub mod turn_timings; @@ -73,7 +74,7 @@ mod tauri_app { question as question_commands, quick_messages as quick_messages_commands, remote_proxy as remote_proxy_commands, remote_workspace as remote_workspace_commands, science as science_commands, - session_info as session_info_commands, + search as search_commands, session_info as session_info_commands, system_settings, terminal as terminal_commands, token_usage as token_usage_commands, version_control, windows, work_task as work_task_commands, @@ -318,6 +319,11 @@ mod tauri_app { // Restore and apply saved system proxy settings before any network operation. let db = app.state::(); + let search_indexer = crate::search::indexer::MessageSearchIndexer::spawn( + db.conn.clone(), + web::event_bridge::EventEmitter::Tauri(app.handle().clone()), + ); + app.manage(search_indexer); tauri::async_runtime::block_on(network::proxy::init_proxy_from_db(&db.conn)); // Logging phase 2/3: override the default level from the @@ -972,6 +978,9 @@ mod tauri_app { conversations::get_conversation, conversations::list_all_conversations, conversations::list_child_conversations, + search_commands::search_conversations, + search_commands::get_search_index_status, + search_commands::set_search_settings, conversations::list_opened_tabs, conversations::save_opened_tabs, conversations::import_local_conversations, diff --git a/src-tauri/src/models/mod.rs b/src-tauri/src/models/mod.rs index 1c860e73f..c31b5dc6e 100644 --- a/src-tauri/src/models/mod.rs +++ b/src-tauri/src/models/mod.rs @@ -9,6 +9,7 @@ pub mod model_provider; pub mod pet; pub mod quick_message; pub mod remote_workspace_connection; +pub mod search; pub mod system; pub mod token_usage; pub mod work_task; @@ -36,6 +37,9 @@ pub use message::{ }; pub use quick_message::QuickMessageInfo; pub use remote_workspace_connection::RemoteWorkspaceConnectionInfo; +pub use search::{ + DbConversationSearchResult, SearchIndexStatus, SearchMatchLocation, SearchMatchLocationKind, +}; pub use token_usage::{ TokenUsageBreakdownItem, TokenUsageBucket, TokenUsageConversationItem, TokenUsageFacets, TokenUsageFilter, TokenUsageFolderFacet, TokenUsageHeatCell, TokenUsagePoint, diff --git a/src-tauri/src/models/search.rs b/src-tauri/src/models/search.rs new file mode 100644 index 000000000..9c8ac0e01 --- /dev/null +++ b/src-tauri/src/models/search.rs @@ -0,0 +1,43 @@ +use serde::Serialize; + +use super::conversation::DbConversationSummary; + +/// Where one search term matches inside a conversation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum SearchMatchLocationKind { + Title, + Content, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct SearchMatchLocation { + pub kind: SearchMatchLocationKind, + pub turn_id: Option, + pub block_index: Option, + pub char_start: usize, + pub char_end: usize, +} + +/// One conversation-level search result. Snippet fields are raw text windows +/// around the first match and are safe to render as text. +#[derive(Clone, Debug, Serialize)] +pub struct DbConversationSearchResult { + pub summary: DbConversationSummary, + pub snippet_prefix: Option, + pub snippet_match: Option, + pub snippet_suffix: Option, + pub matches: Vec, +} + +/// Read-only view of the content index lifecycle for the search dialog. +#[derive(Clone, Debug, Serialize)] +pub struct SearchIndexStatus { + pub mode: String, + pub user_enabled: bool, + pub user_mode: String, + pub indexed_conversation_count: i32, + pub visible_conversation_count: i64, + pub building: bool, + pub progress: f64, +} diff --git a/src-tauri/src/search/indexer.rs b/src-tauri/src/search/indexer.rs new file mode 100644 index 000000000..d3156eea5 --- /dev/null +++ b/src-tauri/src/search/indexer.rs @@ -0,0 +1,558 @@ +use std::collections::HashSet; +use std::sync::Arc; + +use sea_orm::{ColumnTrait, ConnectionTrait, DatabaseConnection, DbBackend, EntityTrait, PaginatorTrait, QueryFilter, Statement}; +use tokio::sync::{mpsc, Mutex}; + +use crate::db::entities::message_search_document; +use crate::db::service::message_search_service::{self, SyncFlags, MODE_FTS, MODE_SCAN}; +use crate::models::{MessageTurn, SearchIndexStatus}; +use crate::search::normalizer::normalize_turns; +use crate::web::event_bridge::{emit_event, EventEmitter, SEARCH_INDEX_PROGRESS_EVENT}; + +const DRIFT_INTERVAL_SECS: u64 = 10 * 60; + +#[derive(Debug)] +enum IndexRequest { + Parse(i32), + Turns { + conversation_id: i32, + turns: Arc>, + }, + Delete(i32), + SyncMode, +} + +pub struct MessageSearchIndexer { + sender: mpsc::UnboundedSender, +} + +impl MessageSearchIndexer { + pub fn spawn(conn: DatabaseConnection, emitter: EventEmitter) -> Arc { + let (sender, receiver) = mpsc::unbounded_channel(); + let handle = Arc::new(Self { sender }); + let pending = Arc::new(Mutex::new(HashSet::new())); + let worker_handle = Arc::clone(&handle); + #[cfg(feature = "tauri-runtime")] + tauri::async_runtime::spawn(async move { + run_worker(conn, emitter, receiver, pending, worker_handle).await; + }); + #[cfg(not(feature = "tauri-runtime"))] + tokio::spawn(async move { + run_worker(conn, emitter, receiver, pending, worker_handle).await; + }); + handle + } + + pub fn request_parse(&self, conversation_id: i32) { + let _ = self.sender.send(IndexRequest::Parse(conversation_id)); + } + + /// Submit already-parsed turns without touching the indexer's queue with a + /// full deep copy: callers create one shared allocation and the worker + /// borrows it. + pub fn submit_turns(&self, conversation_id: i32, turns: Arc>) { + let _ = self.sender.send(IndexRequest::Turns { + conversation_id, + turns, + }); + } + + pub fn request_delete(&self, conversation_id: i32) { + let _ = self.sender.send(IndexRequest::Delete(conversation_id)); + } + + /// Ask the worker to re-evaluate mode / user settings asynchronously so a + /// settings save never blocks the command thread on a full FTS rebuild. + pub fn request_mode_sync(&self) { + let _ = self.sender.send(IndexRequest::SyncMode); + } +} + +async fn run_worker( + conn: DatabaseConnection, + emitter: EventEmitter, + mut receiver: mpsc::UnboundedReceiver, + pending: Arc>>, + handle: Arc, +) { + if let Err(error) = message_search_service::ensure_search_state(&conn).await { + tracing::error!("[search-index] failed to ensure state: {error}"); + } + if let Err(error) = message_search_service::upgrade_schema_if_needed(&conn).await { + tracing::error!("[search-index] schema upgrade failed: {error}"); + } + if let Err(error) = drift_resync(&conn, &handle).await { + tracing::error!("[search-index] initial drift resync failed: {error}"); + } + + let mut drift = tokio::time::interval(std::time::Duration::from_secs(DRIFT_INTERVAL_SECS)); + drift.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + loop { + tokio::select! { + request = receiver.recv() => { + let Some(request) = request else { break; }; + match request { + IndexRequest::Parse(conversation_id) => { + pending.lock().await.insert(conversation_id); + if let Err(error) = process_parse(&conn, conversation_id).await { + tracing::warn!("[search-index] parse {conversation_id} failed: {error}"); + } + pending.lock().await.remove(&conversation_id); + } + IndexRequest::Turns { conversation_id, turns } => { + pending.lock().await.insert(conversation_id); + if let Err(error) = process_turns(&conn, conversation_id, &turns).await { + tracing::warn!("[search-index] turns {conversation_id} failed: {error}"); + } + pending.lock().await.remove(&conversation_id); + } + IndexRequest::Delete(conversation_id) => { + pending.lock().await.insert(conversation_id); + if let Err(error) = process_delete(&conn, conversation_id).await { + tracing::warn!("[search-index] delete {conversation_id} failed: {error}"); + } + pending.lock().await.remove(&conversation_id); + } + IndexRequest::SyncMode => { + if let Err(error) = drift_resync(&conn, &handle).await { + tracing::warn!("[search-index] requested resync failed: {error}"); + } + if let Err(error) = sync_mode_and_progress(&conn, &emitter).await { + tracing::warn!("[search-index] requested mode sync failed: {error}"); + } + } + } + // Sync once after the queued batch drains instead of after every + // item: `pending` is populated by the arms above. + if pending.lock().await.is_empty() { + if let Err(error) = sync_mode_and_progress(&conn, &emitter).await { + tracing::warn!("[search-index] mode sync failed: {error}"); + } + } + } + _ = drift.tick() => { + if let Err(error) = drift_resync(&conn, &handle).await { + tracing::error!("[search-index] drift resync failed: {error}"); + } + if let Err(error) = sync_mode_and_progress(&conn, &emitter).await { + tracing::error!("[search-index] progress sync failed: {error}"); + } + } + } + } +} + +/// Queue parses for every visible conversation whose indexed document is stale +/// or missing, using one LEFT JOIN instead of an N+1 per-row SELECT. The +/// joined metadata is the same probe `process_turns` records, so unchanged +/// conversations are skipped without re-reading their transcripts. +async fn drift_resync( + conn: &DatabaseConnection, + handle: &MessageSearchIndexer, +) -> Result<(), crate::db::error::DbError> { + let state = message_search_service::ensure_search_state(conn).await?; + if !state.user_enabled { + return Ok(()); + } + let rows = conn + .query_all(Statement::from_string( + DbBackend::Sqlite, + "SELECT c.id, c.updated_at, c.message_count, \ + d.source_ended_at, d.source_message_count \ + FROM conversation c \ + LEFT JOIN message_search_document d ON d.conversation_id = c.id \ + WHERE c.deleted_at IS NULL AND c.kind != 'loop' \ + AND c.parent_id IS NULL AND c.external_id IS NOT NULL \ + ORDER BY c.updated_at DESC" + .to_string(), + )) + .await?; + for row in rows { + let conversation_id = row.try_get_by_index::(0)?; + let updated_at = row.try_get_by_index::>(1)?; + let message_count = row.try_get_by_index::(2)?; + let source_ended_at = + row.try_get_by_index::>>(3)?; + let source_message_count = row.try_get_by_index::>(4)?; + let dirty = source_ended_at != Some(updated_at) + || source_message_count != Some(message_count); + if dirty { + handle.request_parse(conversation_id); + } + } + Ok(()) +} + +async fn process_parse( + conn: &DatabaseConnection, + conversation_id: i32, +) -> Result<(), crate::db::error::DbError> { + let (detail, _) = + crate::commands::conversations::get_folder_conversation_core(conn, conversation_id) + .await + .map_err(|error| { + crate::db::error::DbError::Migration(format!("parse conversation: {error}")) + })?; + process_turns(conn, conversation_id, &detail.turns).await +} + +async fn process_turns( + conn: &DatabaseConnection, + conversation_id: i32, + turns: &[MessageTurn], +) -> Result<(), crate::db::error::DbError> { + let state = message_search_service::get_search_state(conn).await?; + if !state.user_enabled { + return Ok(()); + } + let existing = message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.eq(conversation_id)) + .one(conn) + .await?; + let conversation = crate::db::entities::conversation::Entity::find_by_id(conversation_id) + .one(conn) + .await? + .filter(|model| model.deleted_at.is_none()); + let Some(conversation) = conversation else { + return Ok(()); + }; + let source_ended_at = Some(conversation.updated_at); + let source_message_count = conversation.message_count; + // The cheap source-metadata guard mirrors `drift_resync`'s dirtiness probe: + // an unchanged detail load never re-normalizes the whole transcript. + if existing.as_ref().is_some_and(|doc| { + doc.source_ended_at == source_ended_at + && doc.source_message_count == source_message_count + }) { + return Ok(()); + } + + let document = normalize_turns(turns); + // Empty turns (typically a transcript whose session file no longer + // exists) still record a tombstone row so the progress denominator treats + // the conversation as handled. `drift_resync` sees a non-dirty document + // and stops re-queueing the parse every ten minutes. + let sync = sync_flags_for(&state); + if existing.as_ref().is_some_and(|doc| { + doc.content_hash == document.content_hash + && doc.source_ended_at == source_ended_at + && doc.source_message_count == source_message_count + }) { + return Ok(()); + } + message_search_service::upsert_document( + conn, + conversation_id, + &document, + source_ended_at, + source_message_count, + sync, + ) + .await?; + Ok(()) +} + +async fn process_delete( + conn: &DatabaseConnection, + conversation_id: i32, +) -> Result<(), crate::db::error::DbError> { + let state = message_search_service::get_search_state(conn).await?; + let sync = sync_flags_for(&state); + message_search_service::delete_document(conn, conversation_id, sync).await +} + +fn sync_flags_for(state: &crate::db::entities::search_index_state::Model) -> SyncFlags { + SyncFlags { + trigram: state.mode == MODE_FTS, + short: state.mode == MODE_FTS && state.short_fts_enabled, + } +} + +pub(crate) async fn sync_mode_and_progress( + conn: &DatabaseConnection, + emitter: &EventEmitter, +) -> Result<(), crate::db::error::DbError> { + let state = message_search_service::ensure_search_state(conn).await?; + + // Disabled means "do not store my content": drop existing documents and + // report a settled (not building) status rather than a permanent 0%. + if !state.user_enabled { + message_search_service::clear_all_documents(conn).await?; + let visible = message_search_service::indexable_conversation_count(conn).await?; + message_search_service::set_index_progress(conn, 0, Some(chrono::Utc::now())).await?; + let status = SearchIndexStatus { + mode: state.mode, + user_enabled: false, + user_mode: state.user_mode, + indexed_conversation_count: 0, + visible_conversation_count: visible, + building: false, + progress: 1.0, + }; + emit_event(emitter, SEARCH_INDEX_PROGRESS_EVENT, status); + return Ok(()); + } + + message_search_service::delete_orphan_documents(conn).await?; + let bytes = message_search_service::total_indexed_text_bytes(conn).await? as f64; + let threshold_bytes = state.threshold_mb.max(0.000_001) * 1_000_000.0; + // Real hysteresis: only fall back to scan below half the threshold. The + // previous `_ if mode == FTS && bytes < threshold * 0.5` arm returned the + // same tuple as the fallback, so the switch happened at the threshold. + let (mode, short) = match state.user_mode.as_str() { + "scan" => (MODE_SCAN, false), + "fts" => (MODE_FTS, true), + _ => { + if bytes >= threshold_bytes { + (MODE_FTS, true) + } else if state.mode == MODE_FTS && bytes >= threshold_bytes * 0.5 { + (MODE_FTS, state.short_fts_enabled) + } else { + (MODE_SCAN, false) + } + } + }; + + if state.mode != mode || state.short_fts_enabled != short { + message_search_service::rebuild_fts_and_set_mode(conn, mode, short).await?; + } + + let indexed = message_search_document::Entity::find().count(conn).await? as i32; + let visible = message_search_service::indexable_conversation_count(conn).await?; + message_search_service::set_index_progress(conn, indexed, Some(chrono::Utc::now())).await?; + let progress = if visible > 0 { + (indexed as f64 / visible as f64).clamp(0.0, 1.0) + } else { + 1.0 + }; + let status = SearchIndexStatus { + mode: mode.to_string(), + user_enabled: state.user_enabled, + user_mode: state.user_mode, + indexed_conversation_count: indexed, + visible_conversation_count: visible, + building: false, + progress, + }; + emit_event(emitter, SEARCH_INDEX_PROGRESS_EVENT, status); + Ok(()) +} +#[cfg(test)] +mod tests { + use super::*; + use crate::db::entities::{conversation, message_search_document}; + use crate::db::service::message_search_service::USER_MODE_SCAN; + use crate::db::service::message_search_service::USER_MODE_AUTO; + use crate::db::test_helpers::{fresh_in_memory_db, seed_conversation, seed_folder}; + use crate::models::{ContentBlock, TurnRole}; + use sea_orm::ActiveModelTrait; + + fn turn(text: &str) -> MessageTurn { + MessageTurn { + id: "t1".to_string(), + role: TurnRole::User, + blocks: vec![ContentBlock::Text { + text: text.to_string(), + }], + timestamp: chrono::Utc::now(), + usage: None, + duration_ms: None, + model: None, + completed_at: None, + } + } + + #[tokio::test] + async fn mode_sync_upgrades_fts_when_threshold_crossed() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-indexer").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + let state = message_search_service::get_search_state(&db.conn) + .await + .expect("state"); + let mut active: crate::db::entities::search_index_state::ActiveModel = state.into(); + active.threshold_mb = sea_orm::Set(0.00001); + active.update(&db.conn).await.expect("threshold"); + message_search_service::upsert_document( + &db.conn, + conversation_id, + &normalize_turns(&[turn("会话记录")]), + None, + 1, + SyncFlags::default(), + ) + .await + .expect("doc"); + + sync_mode_and_progress(&db.conn, &EventEmitter::Noop) + .await + .expect("sync"); + let state = message_search_service::get_search_state(&db.conn) + .await + .expect("state"); + assert_eq!(state.mode, MODE_FTS); + assert!(state.short_fts_enabled); + } + + #[tokio::test] + async fn unchanged_turns_skip_a_second_write() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-skip").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + let turns = vec![turn("same text")]; + process_turns(&db.conn, conversation_id, &turns) + .await + .expect("first"); + process_turns(&db.conn, conversation_id, &turns) + .await + .expect("second"); + let docs = + message_search_service::list_documents_by_conversation(&db.conn, &[conversation_id]) + .await + .expect("docs"); + assert_eq!(docs.len(), 1); + } + + #[tokio::test] + async fn empty_turns_leave_a_tombstone_document() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-tombstone").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::ClaudeCode).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + + process_turns(&db.conn, conversation_id, &[]) + .await + .expect("tombstone"); + + let docs = + message_search_service::list_documents_by_conversation(&db.conn, &[conversation_id]) + .await + .expect("docs"); + assert_eq!(docs.len(), 1); + assert!(docs[0].1.is_empty()); + } + + #[tokio::test] + async fn stale_metadata_reparses_and_refreshes_source_fields() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-metadata").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + + process_turns(&db.conn, conversation_id, &[turn("same text")]) + .await + .expect("first"); + let first = message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.eq(conversation_id)) + .one(&db.conn) + .await + .expect("read") + .expect("document"); + + crate::db::service::conversation_service::update_status( + &db.conn, + conversation_id, + conversation::ConversationStatus::Completed, + ) + .await + .expect("bump"); + + process_turns(&db.conn, conversation_id, &[turn("same text")]) + .await + .expect("second"); + let second = message_search_document::Entity::find() + .filter(message_search_document::Column::ConversationId.eq(conversation_id)) + .one(&db.conn) + .await + .expect("read") + .expect("document"); + + assert_eq!(second.content_hash, first.content_hash); + assert_ne!(second.source_ended_at, first.source_ended_at); + assert_eq!(second.source_message_count, first.source_message_count); + } + + #[tokio::test] + async fn scan_user_mode_clears_fts_posting() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-scan-mode").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + message_search_service::set_search_mode(&db.conn, MODE_FTS, true) + .await + .expect("mode"); + message_search_service::upsert_document( + &db.conn, + conversation_id, + &normalize_turns(&[turn("会话")]), + None, + 1, + SyncFlags { + trigram: true, + short: true, + }, + ) + .await + .expect("doc"); + message_search_service::set_search_user_settings(&db.conn, true, USER_MODE_SCAN) + .await + .expect("user mode"); + sync_mode_and_progress(&db.conn, &EventEmitter::Noop) + .await + .expect("sync"); + let state = message_search_service::get_search_state(&db.conn) + .await + .expect("state"); + assert_eq!(state.mode, MODE_SCAN); + } + #[tokio::test] + async fn disabling_search_clears_stored_documents() { + let db = fresh_in_memory_db().await; + let folder_id = seed_folder(&db, "/tmp/search-disable").await; + let conversation_id = + seed_conversation(&db, folder_id, crate::models::AgentType::Codex).await; + message_search_service::ensure_search_state(&db.conn) + .await + .expect("state"); + message_search_service::upsert_document( + &db.conn, + conversation_id, + &normalize_turns(&[turn("secret text")]), + None, + 1, + SyncFlags::default(), + ) + .await + .expect("doc"); + message_search_service::set_search_user_settings(&db.conn, false, USER_MODE_AUTO) + .await + .expect("settings"); + + sync_mode_and_progress(&db.conn, &EventEmitter::Noop) + .await + .expect("sync"); + let count = message_search_document::Entity::find() + .count(&db.conn) + .await + .expect("count"); + assert_eq!(count, 0); + } +} \ No newline at end of file diff --git a/src-tauri/src/search/mod.rs b/src-tauri/src/search/mod.rs new file mode 100644 index 000000000..0fd84296a --- /dev/null +++ b/src-tauri/src/search/mod.rs @@ -0,0 +1,3 @@ +pub mod indexer; +pub mod normalizer; +pub mod query; diff --git a/src-tauri/src/search/normalizer.rs b/src-tauri/src/search/normalizer.rs new file mode 100644 index 000000000..e94e32991 --- /dev/null +++ b/src-tauri/src/search/normalizer.rs @@ -0,0 +1,214 @@ +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use crate::models::{ContentBlock, MessageTurn, TurnRole}; + +/// Maximum UTF-8 bytes kept from one text block. +pub const MAX_BLOCK_BYTES: usize = 8_192; + +/// The normalized, indexable representation of one conversation. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NormalizedDocument { + pub text: String, + pub content_hash: String, + pub blocks: Vec, +} + +/// One indexed text block's position in the normalized document. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct NormalizedBlockOffset { + pub turn_id: String, + pub block_index: usize, + pub start: usize, + pub end: usize, + pub leading_trim: usize, +} + +/// Tokens for the optional short-query FTS table. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ShortIndexTokens { + pub words: String, + pub bigrams: String, +} + +/// Extract only user and assistant `Text` blocks into one searchable document. +/// +/// System prompts, reasoning, tool blocks, and images never reach this +/// function's input in their indexed form; the role/block filters here are the +/// second, defensive boundary. +pub fn normalize_turns(turns: &[MessageTurn]) -> NormalizedDocument { + let mut text = String::new(); + let mut blocks = Vec::new(); + let mut wrote_block = false; + + for turn in turns { + if !matches!(turn.role, TurnRole::User | TurnRole::Assistant) { + continue; + } + for (block_index, block) in turn.blocks.iter().enumerate() { + let ContentBlock::Text { text: block_text } = block else { + continue; + }; + let trimmed = block_text.trim(); + if trimmed.is_empty() { + continue; + } + let leading_trim = block_text.chars().count() - block_text.trim_start().chars().count(); + if wrote_block { + text.push_str("\n\n"); + } + let start = text.chars().count(); + text.push_str(&truncate_text_block(trimmed, MAX_BLOCK_BYTES)); + let end = text.chars().count(); + blocks.push(NormalizedBlockOffset { + turn_id: turn.id.clone(), + block_index, + start, + end, + leading_trim, + }); + wrote_block = true; + } + } + + NormalizedDocument { + content_hash: sha256_hex(&text), + text, + blocks, + } +} + +/// Truncate a UTF-8 string to `max_bytes` without splitting a character. +pub fn truncate_text_block(text: &str, max_bytes: usize) -> String { + if text.len() <= max_bytes { + return text.to_string(); + } + let mut end = max_bytes; + while end > 0 && !text.is_char_boundary(end) { + end -= 1; + } + text[..end].to_string() +} + +/// Lowercase SHA-256 hex digest, used for cheap content diffing. +pub fn sha256_hex(input: &str) -> String { + let digest = Sha256::digest(input.as_bytes()); + format!("{digest:x}") +} + +/// Build `words` and `bigrams` for the short-query FTS index. +pub fn short_index_tokens(text: &str) -> ShortIndexTokens { + let mut words = String::new(); + let mut bigrams = String::new(); + let mut runs: Vec<(bool, Vec)> = Vec::new(); + + for ch in text.chars() { + let cjk = is_cjk(ch); + match runs.last_mut() { + Some((kind, chars)) if *kind == cjk => chars.push(ch), + _ => runs.push((cjk, vec![ch])), + } + } + + for (cjk, chars) in runs { + if cjk { + for &ch in &chars { + push_token(&mut words, &ch.to_string()); + } + for pair in chars.windows(2) { + let token: String = pair.iter().collect(); + push_token(&mut bigrams, &token); + } + } else { + let mut current = String::new(); + for ch in chars { + if ch.is_alphanumeric() { + current.extend(ch.to_lowercase()); + } else if !current.is_empty() { + push_token(&mut words, ¤t); + current.clear(); + } + } + if !current.is_empty() { + push_token(&mut words, ¤t); + } + } + } + + ShortIndexTokens { words, bigrams } +} + +fn push_token(output: &mut String, token: &str) { + if token.is_empty() { + return; + } + if !output.is_empty() { + output.push(' '); + } + output.push_str(token); +} + +/// CJK ranges covered by the short-query index. Fullwidth Latin, symbols, and +/// other scripts intentionally stay on the Latin/word path. +pub fn is_cjk(ch: char) -> bool { + matches!( + ch as u32, + 0x3400..=0x4DBF + | 0x4E00..=0x9FFF + | 0xF900..=0xFAFF + | 0x20000..=0x2FA1F + | 0x3040..=0x30FF + | 0xAC00..=0xD7AF + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::models::MessageTurn; + + fn text_turn(role: TurnRole, text: &str) -> MessageTurn { + MessageTurn { + id: "turn-1".to_string(), + role, + blocks: vec![ContentBlock::Text { + text: text.to_string(), + }], + timestamp: chrono::Utc::now(), + usage: None, + duration_ms: None, + model: None, + completed_at: None, + } + } + + #[test] + fn normalizes_only_user_and_assistant_text() { + let turns = vec![ + text_turn(TurnRole::User, " 你好 "), + text_turn(TurnRole::Assistant, " 回答 "), + text_turn(TurnRole::System, " 系统 "), + ]; + let doc = normalize_turns(&turns); + assert_eq!(doc.text, "你好\n\n回答"); + assert_eq!(doc.content_hash.len(), 64); + assert_eq!(doc.blocks.len(), 2); + assert_eq!(doc.blocks[0].leading_trim, 2); + assert_eq!(doc.blocks[1].start, "你好\n\n".chars().count()); + } + + #[test] + fn truncation_never_splits_utf8() { + let long = "你".repeat(4_096) + "a"; + let truncated = truncate_text_block(&long, MAX_BLOCK_BYTES); + assert!(truncated.len() <= MAX_BLOCK_BYTES); + assert_eq!(truncated.chars().last(), Some('你')); + } + + #[test] + fn short_tokens_emit_cjk_unigrams_and_bigrams() { + let tokens = short_index_tokens("搜索chat"); + assert_eq!(tokens.words, "搜 索 chat"); + assert_eq!(tokens.bigrams, "搜索"); + } +} diff --git a/src-tauri/src/search/query.rs b/src-tauri/src/search/query.rs new file mode 100644 index 000000000..a7506b808 --- /dev/null +++ b/src-tauri/src/search/query.rs @@ -0,0 +1,145 @@ +/// A single user query is split into at most this many terms. +pub const MAX_QUERY_TERMS: usize = 8; + +/// Split a query on Unicode whitespace and drop empty terms. +pub fn split_terms(query: &str) -> Vec { + query + .split_whitespace() + .map(str::trim) + .filter(|term| !term.is_empty()) + .take(MAX_QUERY_TERMS) + .map(ToOwned::to_owned) + .collect() +} + +/// Escape SQLite `LIKE` metacharacters for an `ESCAPE '\'` pattern. +pub fn escape_like(term: &str) -> String { + let mut escaped = String::with_capacity(term.len()); + for ch in term.chars() { + match ch { + '\\' | '%' | '_' => { + escaped.push('\\'); + escaped.push(ch); + } + _ => escaped.push(ch), + } + } + escaped +} + +/// A parameterized substring pattern: `%escaped%`. +pub fn like_pattern(term: &str) -> String { + format!("%{}%", escape_like(term)) +} + +/// Build an FTS5 trigram `MATCH` expression for terms of at least three +/// Unicode characters. Adjacent three-character grams are ANDed; the caller +/// still applies `LIKE` as the exact substring filter. +pub fn trigram_expression(term: &str) -> Option { + let chars: Vec = term.chars().collect(); + if chars.len() < 3 { + return None; + } + let grams = chars + .windows(3) + .map(|window| format!("\"{}\"", fts_quote(&window.iter().collect::()))) + .collect::>() + .join(" AND "); + Some(grams) +} + +/// How a one- or two-character term is queried against the short FTS table. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ShortTermQuery { + CjkUnigram { token: String }, + CjkBigram { phrase: String }, + LatinPrefix { token: String }, +} + +/// Build a short-term query. Callers use this only when the term is one or two +/// Unicode characters; longer terms use the trigram index. +pub fn short_query(term: &str) -> ShortTermQuery { + let chars: Vec = term.chars().collect(); + let all_cjk = !chars.is_empty() + && chars + .iter() + .all(|ch| crate::search::normalizer::is_cjk(*ch)); + if all_cjk { + if chars.len() == 1 { + ShortTermQuery::CjkUnigram { + token: fts_quote(&chars[0].to_string()), + } + } else { + let bigrams: Vec = chars + .windows(2) + .map(|window| fts_quote(&window.iter().collect::())) + .collect(); + ShortTermQuery::CjkBigram { + phrase: bigrams.join(" "), + } + } + } else { + ShortTermQuery::LatinPrefix { + token: fts_quote(&term.to_lowercase()), + } + } +} + +/// Quote an FTS5 phrase element and double embedded quotes. +pub fn fts_quote(input: &str) -> String { + input.replace('"', "\"\"") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn splits_terms_and_caps_at_eight() { + let terms = split_terms(" one 两个\tthree four five six seven eight nine ten "); + assert_eq!(terms.len(), 8); + assert_eq!(terms[0], "one"); + assert_eq!(terms[1], "两个"); + } + + #[test] + fn escapes_like_metacharacters() { + assert_eq!(escape_like(r"100%_ok\"), r"100\%\_ok\\"); + assert_eq!(like_pattern("a%b"), "%a\\%b%"); + } + + #[test] + fn trigram_expression_needs_three_chars_and_quotes() { + assert_eq!(trigram_expression("ab"), None); + assert_eq!( + trigram_expression("会话记录"), + Some("\"会话记\" AND \"话记录\"".to_string()) + ); + assert_eq!( + trigram_expression("a\"bcd"), + Some("\"a\"\"b\" AND \"\"\"bc\" AND \"bcd\"".to_string()) + ); + } + + #[test] + fn routes_short_queries_by_script() { + assert_eq!( + short_query("会"), + ShortTermQuery::CjkUnigram { + token: "会".to_string() + } + ); + assert_eq!( + short_query("聊天"), + ShortTermQuery::CjkBigram { + phrase: "聊天".to_string() + } + ); + assert_eq!( + short_query("Ab"), + ShortTermQuery::LatinPrefix { + token: "ab".to_string() + } + ); + } +} diff --git a/src-tauri/src/web/event_bridge.rs b/src-tauri/src/web/event_bridge.rs index 52dc5ef74..d43b0e9b0 100644 --- a/src-tauri/src/web/event_bridge.rs +++ b/src-tauri/src/web/event_bridge.rs @@ -149,6 +149,10 @@ impl EventEmitter { /// Global side-channel for cross-client conversation list/status sync. pub const CONVERSATION_CHANGED_EVENT: &str = "conversation://changed"; +/// Global side-channel for the content-search indexer's progress snapshot. +/// Payload: `crate::models::SearchIndexStatus`. +pub const SEARCH_INDEX_PROGRESS_EVENT: &str = "search-index://progress"; + /// Global side-channel announcing a live-feedback enable/disable. The settings /// UI runs in a SEPARATE window (`openSettingsWindow`), so the conversation /// feedback bar can't learn about a save through any frontend-only cache — it diff --git a/src-tauri/src/web/handlers/conversations.rs b/src-tauri/src/web/handlers/conversations.rs index a46a20fc1..1e7eca174 100644 --- a/src-tauri/src/web/handlers/conversations.rs +++ b/src-tauri/src/web/handlers/conversations.rs @@ -143,15 +143,20 @@ pub async fn get_folder_conversation( ) -> Result, AppCommandError> { let db = &state.db; let window = conv_commands::resolve_turn_window_req(params.tail_turns, params.from_index)?; - let result = conv_commands::get_folder_conversation_with_live_core( + let mut result = conv_commands::get_folder_conversation_with_live_core( &db.conn, &state.connection_manager, &state.chat_channel_manager, &state.emitter, params.conversation_id, - window, ) .await?; + if let Some(indexer) = &state.search_indexer { + indexer.submit_turns(params.conversation_id, std::sync::Arc::new(result.turns.clone())); + } + if let Some(req) = window { + conv_commands::apply_turn_window(&mut result, req); + } Ok(Json(result)) } @@ -202,28 +207,36 @@ pub async fn import_local_conversations( Extension(state): Extension>, Json(params): Json, ) -> Result, AppCommandError> { - Ok(Json( - conv_commands::import_local_conversations_core( - &state.db.conn, - &state.emitter, - &state.chat_channel_manager, - params.folder_id, - ) - .await?, - )) + let (result, updated_ids) = conv_commands::import_local_conversations_core( + &state.db.conn, + &state.emitter, + &state.chat_channel_manager, + params.folder_id, + ) + .await?; + if let Some(indexer) = &state.search_indexer { + for id in updated_ids { + indexer.request_parse(id); + } + } + Ok(Json(result)) } pub async fn scan_importable_sessions( Extension(state): Extension>, ) -> Result, AppCommandError> { - Ok(Json( - conv_commands::scan_importable_sessions_core( - &state.db.conn, - &state.emitter, - &state.chat_channel_manager, - ) - .await?, - )) + let (result, refreshed_ids) = conv_commands::scan_importable_sessions_core( + &state.db.conn, + &state.emitter, + &state.chat_channel_manager, + ) + .await?; + if let Some(indexer) = &state.search_indexer { + for id in refreshed_ids { + indexer.request_parse(id); + } + } + Ok(Json(result)) } #[derive(Deserialize)] @@ -395,5 +408,8 @@ pub async fn delete_conversation( params.conversation_id, ) .await?; + if let Some(indexer) = &state.search_indexer { + indexer.request_delete(params.conversation_id); + } Ok(Json(())) } diff --git a/src-tauri/src/web/handlers/mod.rs b/src-tauri/src/web/handlers/mod.rs index 045d1e168..229d456e6 100644 --- a/src-tauri/src/web/handlers/mod.rs +++ b/src-tauri/src/web/handlers/mod.rs @@ -27,6 +27,7 @@ pub mod project_boot; pub mod question; pub mod quick_messages; pub mod science; +pub mod search; pub mod session_info; pub mod system_settings; pub mod terminal; diff --git a/src-tauri/src/web/handlers/search.rs b/src-tauri/src/web/handlers/search.rs new file mode 100644 index 000000000..139c099fa --- /dev/null +++ b/src-tauri/src/web/handlers/search.rs @@ -0,0 +1,61 @@ +use std::sync::Arc; + +use axum::{extract::Extension, Json}; +use serde::Deserialize; + +use crate::app_error::AppCommandError; +use crate::app_state::AppState; +use crate::commands::search as search_commands; +use crate::models::{AgentType, DbConversationSearchResult, SearchIndexStatus}; + +#[derive(Deserialize, Default)] +#[serde(rename_all = "camelCase")] +pub struct SearchConversationsParams { + pub folder_ids: Option>, + pub agent_type: Option, + pub query: String, + pub limit: Option, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct SetSearchSettingsParams { + pub enabled: bool, + pub user_mode: String, +} + +pub async fn search_conversations( + Extension(state): Extension>, + Json(params): Json, +) -> Result>, AppCommandError> { + Ok(Json( + search_commands::search_conversations_core( + &state.db.conn, + params.folder_ids, + params.agent_type, + params.query, + params.limit, + ) + .await?, + )) +} + +pub async fn get_search_index_status( + Extension(state): Extension>, +) -> Result, AppCommandError> { + Ok(Json( + search_commands::get_search_index_status_core(&state.db.conn).await?, + )) +} + +pub async fn set_search_settings( + Extension(state): Extension>, + Json(params): Json, +) -> Result, AppCommandError> { + search_commands::set_search_settings_core(&state.db.conn, params.enabled, params.user_mode) + .await?; + if let Some(indexer) = &state.search_indexer { + indexer.request_mode_sync(); + } + Ok(Json(())) +} diff --git a/src-tauri/src/web/mod.rs b/src-tauri/src/web/mod.rs index ec2682db9..795786c3d 100644 --- a/src-tauri/src/web/mod.rs +++ b/src-tauri/src/web/mod.rs @@ -853,6 +853,9 @@ pub(crate) async fn do_start_web_server_tauri( .state::() .inner() .clone(), + search_indexer: app + .try_state::>() + .map(|state| state.inner().clone()), }); // See do_start_web_server_with_state for rationale on the reset. diff --git a/src-tauri/src/web/router.rs b/src-tauri/src/web/router.rs index faae6d2cb..913b158ac 100644 --- a/src-tauri/src/web/router.rs +++ b/src-tauri/src/web/router.rs @@ -53,6 +53,18 @@ pub fn build_router( "/list_all_conversations", post(handlers::conversations::list_all_conversations), ) + .route( + "/search_conversations", + post(handlers::search::search_conversations), + ) + .route( + "/get_search_index_status", + post(handlers::search::get_search_index_status), + ) + .route( + "/set_search_settings", + post(handlers::search::set_search_settings), + ) .route( "/list_child_conversations", post(handlers::conversations::list_child_conversations), diff --git a/src/app/globals.css b/src/app/globals.css index 8e2e58466..de207df41 100644 --- a/src/app/globals.css +++ b/src/app/globals.css @@ -2553,3 +2553,29 @@ div.monaco-diff-editor.side-by-side .editor.modified { --tu-seq-6: color-mix(in oklab, var(--tu-accent) 81%, var(--card)); --tu-seq-7: var(--tu-accent); } + +/* Search-hit flash used by the Ctrl+K jump. The title mark is rendered by + React; the message body uses CSS.highlights so the DOM is never mutated. + A dedicated token keeps the yellow legible in both themes. */ +:root { + --search-hit-bg: #fde047; +} + +.dark { + --search-hit-bg: rgb(250 204 21 / 0.55); +} + +@keyframes codeg-search-flash { + 0% { + background-color: var(--search-hit-bg); + } + 100% { + background-color: transparent; + } +} + +[data-search-flash] { + padding: 0; + color: inherit; + animation: codeg-search-flash 1.8s ease-out forwards; +} diff --git a/src/components/conversations/conversation-detail-header.tsx b/src/components/conversations/conversation-detail-header.tsx index 10c8f5182..eab44c172 100644 --- a/src/components/conversations/conversation-detail-header.tsx +++ b/src/components/conversations/conversation-detail-header.tsx @@ -1,6 +1,6 @@ "use client" -import { memo, useCallback, useState } from "react" +import { memo, useCallback, useMemo, useState } from "react" import { ChevronRight, Circle, @@ -25,7 +25,7 @@ import { ConversationHeaderFolderPicker } from "@/components/chat/conversation-c import { useAppWorkspaceStore } from "@/stores/app-workspace-store" import { useTabActions } from "@/contexts/tab-context" import { getRuntimeSession } from "@/stores/conversation-runtime-store" -import type { ConversationStatus } from "@/lib/types" +import type { ConversationStatus, SearchMatchLocation } from "@/lib/types" import { STATUS_ORDER } from "@/lib/types" import { ConversationStatusDot } from "@/components/conversations/conversation-status-dot" import { @@ -75,6 +75,64 @@ interface ConversationDetailHeaderProps { folderPath: string | undefined title: string status: ConversationStatus | undefined + searchMatch?: SearchMatchLocation | null + searchQuery?: string | null +} + +function HighlightedSearchTitle({ + title, + range, + flashKey, +}: { + title: string + range: { start: number; end: number } | null + flashKey: string +}) { + if (!range) return <>{title} + const chars = Array.from(title) + const start = Math.min(Math.max(range.start, 0), chars.length) + const end = Math.min(Math.max(range.end, start), chars.length) + if (start >= end) return <>{title} + return ( + <> + {chars.slice(0, start).join("")} + + {chars.slice(start, end).join("")} + + {chars.slice(end).join("")} + + ) +} + +function findDisplayTitleRange( + title: string, + match: SearchMatchLocation | null, + query: string | null +): { start: number; end: number } | null { + const trimmedQuery = query?.trim() + if (trimmedQuery) { + const titleChars = Array.from(title.toLocaleLowerCase()) + const queryChars = Array.from(trimmedQuery.toLocaleLowerCase()) + if (queryChars.length === 0 || queryChars.length > titleChars.length) { + return null + } + for ( + let start = 0; + start <= titleChars.length - queryChars.length; + start++ + ) { + if ( + queryChars.every((char, index) => char === titleChars[start + index]) + ) { + return { start, end: start + queryChars.length } + } + } + } + if (!match || match.kind !== "title") return null + return { + start: match.char_start, + end: match.char_end, + } } /** @@ -99,6 +157,8 @@ export const ConversationDetailHeader = memo(function ConversationDetailHeader({ folderPath, title, status, + searchMatch = null, + searchQuery = null, }: ConversationDetailHeaderProps) { const t = useTranslations("Folder.conversationCard") const ime = useImeGuard() @@ -147,6 +207,10 @@ export const ConversationDetailHeader = memo(function ConversationDetailHeader({ const persisted = conversationId != null const displayTitle = formatConversationTitle(title) || t("untitledConversation") + const titleRange = useMemo( + () => findDisplayTitleRange(displayTitle, searchMatch, searchQuery), + [displayTitle, searchMatch, searchQuery] + ) const handleTogglePin = useCallback(() => { if (conversationId == null) return @@ -256,7 +320,11 @@ export const ConversationDetailHeader = memo(function ConversationDetailHeader({ className="min-w-0 flex-1 truncate text-sm text-foreground/90" title={title} > - {displayTitle} +
diff --git a/src/components/conversations/conversation-detail-panel.tsx b/src/components/conversations/conversation-detail-panel.tsx index 276efbbe9..224ffb9d3 100644 --- a/src/components/conversations/conversation-detail-panel.tsx +++ b/src/components/conversations/conversation-detail-panel.tsx @@ -99,6 +99,7 @@ import { } from "@/stores/conversation-runtime-store" import { useShallow } from "zustand/react/shallow" import { useConversationDetail } from "@/hooks/use-conversation-detail" +import { useSearchFocusStore } from "@/stores/search-focus-store" import { extractUserImagesFromDraft, getPromptDraftDisplayText, @@ -246,6 +247,17 @@ const ConversationTabView = memo(function ConversationTabView({ const tDiag = useTranslations("DiagnosticsSettings") const sharedT = useTranslations("Folder.chat.shared") const tMessageList = useTranslations("Folder.chat.messageList") + const searchFocus = useSearchFocusStore((s) => s.focus) + const activeSearchMatch = useMemo(() => { + if ( + !searchFocus || + searchFocus.conversationId !== conversationId || + searchFocus.contentMatches.length === 0 + ) { + return null + } + return searchFocus.contentMatches[searchFocus.activeMatchIndex] ?? null + }, [conversationId, searchFocus]) const refreshConversations = useAppWorkspaceStore( (s) => s.refreshConversations ) @@ -1726,6 +1738,15 @@ const ConversationTabView = memo(function ConversationTabView({ [acpActions, tabId] ) + const handleNextSearchMatch = useCallback(() => { + useSearchFocusStore.getState().advance() + }, []) + const handleSearchNavigationFailed = useCallback(() => { + if (searchFocus?.conversationId === conversationId) { + useSearchFocusStore.getState().clear() + } + }, [conversationId, searchFocus]) + const messageListNode = ( 0 + ? searchFocus.activeMatchIndex + 1 + : null + } + searchMatchTotal={searchFocus?.contentMatches.length ?? null} + onNextMatch={handleNextSearchMatch} + onSearchNavigationFailed={handleSearchNavigationFailed} /> ) @@ -2079,6 +2110,7 @@ function SplitStripCornerReserve({ side }: { side: "left" | "right" }) { export function ConversationDetailPanel() { const t = useTranslations("Folder.conversation") const tDetails = useTranslations("Folder.sessionDetails") + const searchFocus = useSearchFocusStore((s) => s.focus) const { completeTurn: runtimeCompleteTurn, removeConversation: runtimeRemoveConversation, @@ -2602,6 +2634,16 @@ export function ConversationDetailPanel() { folderPath={selTabFolder?.path} title={selTab.title} status={selTab.status as ConversationStatus | undefined} + searchMatch={ + searchFocus?.conversationId === selTab.conversationId + ? (searchFocus.titleMatches[0] ?? null) + : null + } + searchQuery={ + searchFocus?.conversationId === selTab.conversationId + ? searchFocus.query + : null + } />
)} @@ -2650,6 +2692,16 @@ export function ConversationDetailPanel() { folderPath={activeTabFolder?.path} title={activeTab.title} status={activeTab.status as ConversationStatus | undefined} + searchMatch={ + searchFocus?.conversationId === activeTab.conversationId + ? (searchFocus.titleMatches[0] ?? null) + : null + } + searchQuery={ + searchFocus?.conversationId === activeTab.conversationId + ? searchFocus.query + : null + } /> )} diff --git a/src/components/conversations/search-command-dialog.tsx b/src/components/conversations/search-command-dialog.tsx index 04d0467f6..3776135df 100644 --- a/src/components/conversations/search-command-dialog.tsx +++ b/src/components/conversations/search-command-dialog.tsx @@ -11,11 +11,12 @@ import { useAppWorkspaceStore } from "@/stores/app-workspace-store" import { useTabActions } from "@/contexts/tab-context" import { useWorkbenchRoute } from "@/contexts/workbench-route-context" import { useWorkspaceActions } from "@/contexts/workspace-context" -import { listAllConversations } from "@/lib/api" +import { getSearchIndexStatus, searchConversations } from "@/lib/api" import type { AgentType, ConversationStatus, - DbConversationSummary, + DbConversationSearchResult, + SearchIndexStatus, } from "@/lib/types" import { useFileTree, type FlatFileEntry } from "@/hooks/use-file-tree" import { rankFileMatches } from "@/lib/file-search-match" @@ -31,8 +32,16 @@ import { CommandGroup, CommandItem, } from "@/components/ui/command" +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select" import { cn } from "@/lib/utils" import { formatConversationTitle } from "@/lib/conversation-title" +import { useSearchFocusStore } from "@/stores/search-focus-store" type SearchTab = "conversations" | "files" @@ -49,15 +58,28 @@ export function SearchCommandDialog({ const locale = useLocale() const dateFnsLocale = locale === "zh-CN" ? zhCN : locale === "zh-TW" ? zhTW : enUS - const { activeFolder: folder, activeFolderId } = useActiveFolder() + const { activeFolder } = useActiveFolder() + const allFolders = useAppWorkspaceStore((s) => s.allFolders) const allConversations = useAppWorkspaceStore((s) => s.conversations) - const folderId = activeFolderId ?? 0 - const conversations = useMemo( + const [scopeFolderId, setScopeFolderId] = useState(null) + const selectableFolders = useMemo( + () => allFolders.filter((folder) => folder.kind !== "chat"), + [allFolders] + ) + const scopeFolder = useMemo( + () => + scopeFolderId == null + ? null + : (selectableFolders.find((folder) => folder.id === scopeFolderId) ?? + null), + [selectableFolders, scopeFolderId] + ) + const scopedConversations = useMemo( () => - activeFolderId == null - ? [] - : allConversations.filter((c) => c.folder_id === activeFolderId), - [allConversations, activeFolderId] + scopeFolderId == null + ? allConversations + : allConversations.filter((c) => c.folder_id === scopeFolderId), + [allConversations, scopeFolderId] ) const { openTab } = useTabActions() const { openConversations } = useWorkbenchRoute() @@ -67,11 +89,15 @@ export function SearchCommandDialog({ const [activeTab, setActiveTab] = useState("conversations") const [query, setQuery] = useState("") const [agentFilter, setAgentFilter] = useState(null) - const [results, setResults] = useState([]) + const [results, setResults] = useState([]) const [searching, setSearching] = useState(false) + const [indexStatus, setIndexStatus] = useState(null) const debounceRef = useRef>(undefined) + const searchGenerationRef = useRef(0) + const resultsQueryRef = useRef("") + const searchInputRef = useRef(null) - const folderPath = folder?.path ?? "" + const folderPath = activeFolder?.path ?? "" // File search via shared hook (lazy-loaded when files tab is active) const { @@ -85,7 +111,7 @@ export function SearchCommandDialog({ // Compute which agent types exist in current folder const availableAgents = Array.from( - new Set(conversations.map((c) => c.agent_type)) + new Set(scopedConversations.map((c) => c.agent_type)) ).sort(compareAgentType) // Rank files by relevance (name/path tiers + fuzzy subsequence), scanning the @@ -97,26 +123,38 @@ export function SearchCommandDialog({ const doSearch = useCallback( async (q: string, agent: AgentType | null) => { + const generation = searchGenerationRef.current + 1 + searchGenerationRef.current = generation if (!q.trim() && !agent) { setResults([]) + resultsQueryRef.current = "" setSearching(false) return } setSearching(true) try { - const data = await listAllConversations({ - folder_ids: folderId > 0 ? [folderId] : null, - search: q.trim() || null, + const data = await searchConversations({ + folder_ids: scopeFolderId == null ? null : [scopeFolderId], + query: q.trim() || "", agent_type: agent, + limit: 50, }) + // A slower older request must never overwrite the newest result set. + if (searchGenerationRef.current !== generation) return setResults(data) + resultsQueryRef.current = q.trim() } catch { - setResults([]) + if (searchGenerationRef.current === generation) { + setResults([]) + resultsQueryRef.current = "" + } } finally { - setSearching(false) + if (searchGenerationRef.current === generation) { + setSearching(false) + } } }, - [folderId] + [scopeFolderId] ) // Debounced search on query change (conversations tab only) @@ -133,25 +171,85 @@ export function SearchCommandDialog({ // Reset state when dialog closes useEffect(() => { + if (open) { + useSearchFocusStore.getState().clear() + } if (!open) { setQuery("") setAgentFilter(null) setResults([]) setActiveTab("conversations") + setScopeFolderId(null) resetFileTree() } }, [open, resetFileTree]) + // cmdk owns keyboard navigation, but the conversation input should own the + // initial focus when the dialog opens (and whenever the user returns to this + // tab). The next animation frame lets Radix/cmdk finish mounting first. + useEffect(() => { + if (!open || activeTab !== "conversations") return + const frame = requestAnimationFrame(() => { + searchInputRef.current?.focus() + }) + return () => cancelAnimationFrame(frame) + }, [open, activeTab]) + + // Poll the background indexer while the dialog is open. + useEffect(() => { + if (!open) { + setIndexStatus(null) + return + } + let cancelled = false + const refresh = () => { + getSearchIndexStatus() + .then((status) => { + if (!cancelled) setIndexStatus(status) + }) + .catch(() => {}) + } + refresh() + const timer = window.setInterval(refresh, 5000) + return () => { + cancelled = true + window.clearInterval(timer) + } + }, [open]) + const handleSelectConversation = useCallback( - (conv: DbConversationSummary) => { + (conv: DbConversationSearchResult) => { // Leave any workbench route (e.g. Automations) so the picked conversation // isn't stranded behind the route overlay — covers re-selecting the // already-active tab, which doesn't change activeTabId. openConversations() - openTab(conv.folder_id, conv.id, conv.agent_type, true) + const occurrenceByTurn = new Map() + const contentMatches = conv.matches + .filter((match) => match.kind === "content") + .map((match) => { + const turnKey = match.turn_id ?? "" + const occurrenceIndex = occurrenceByTurn.get(turnKey) ?? 0 + occurrenceByTurn.set(turnKey, occurrenceIndex + 1) + return { ...match, occurrenceIndex } + }) + // The jump must use the same query that produced these matches, not the + // (possibly newer) text currently in the input. + useSearchFocusStore.getState().setFocus({ + conversationId: conv.summary.id, + query: resultsQueryRef.current || query, + titleMatches: conv.matches.filter((match) => match.kind === "title"), + contentMatches, + activeMatchIndex: 0, + }) + openTab( + conv.summary.folder_id, + conv.summary.id, + conv.summary.agent_type, + true + ) onOpenChange(false) }, - [openTab, onOpenChange, openConversations] + [openTab, onOpenChange, openConversations, query] ) const handleSelectFile = useCallback( @@ -173,24 +271,28 @@ export function SearchCommandDialog({ const placeholder = activeTab === "conversations" ? t("placeholder") : t("filePlaceholder") + const contextFolder = + activeTab === "conversations" ? scopeFolder : activeFolder return ( {/* Folder context header */} - {folder && ( + {contextFolder && (
- {t("dialogTitleWithFolder", { name: folder.name })} + {t("dialogTitleWithFolder", { name: contextFolder.name })}
)} @@ -228,9 +330,35 @@ export function SearchCommandDialog({ { + setScopeFolderId(value === "all" ? null : Number(value)) + }} + > + + + + + {t("scopeAllFolders")} + {selectableFolders.map((folder) => ( + + {folder.name} + + ))} + + + ) : undefined + } /> {/* Agent filter (conversations tab only). Wraps: one chip per agent type @@ -270,6 +398,16 @@ export function SearchCommandDialog({ )} + {activeTab === "conversations" && + indexStatus && + indexStatus.progress < 1 && ( +
+ {t("indexing", { + progress: Math.round(indexStatus.progress * 100), + })} +
+ )} + {/* Conversations tab */} {activeTab === "conversations" && ( @@ -285,22 +423,35 @@ export function SearchCommandDialog({ {results.map((conv) => ( handleSelectConversation(conv)} > - - {formatConversationTitle(conv.title) || - t("untitledConversation")} - +
+ + {formatConversationTitle(conv.summary.title) || + t("untitledConversation")} + + {conv.snippet_match && ( + + {conv.snippet_prefix} + + {conv.snippet_match} + + {conv.snippet_suffix} + + )} +
- {getAgentLabel(conv.agent_type)} + {getAgentLabel(conv.summary.agent_type)} - {formatDistanceToNow(new Date(conv.created_at), { + {formatDistanceToNow(new Date(conv.summary.created_at), { addSuffix: true, locale: dateFnsLocale, })} diff --git a/src/components/message/message-list-view.tsx b/src/components/message/message-list-view.tsx index cdaf46d5d..e1216358e 100644 --- a/src/components/message/message-list-view.tsx +++ b/src/components/message/message-list-view.tsx @@ -61,7 +61,12 @@ import { buildPlanKey, extractLatestPlanEntriesFromMessages, } from "@/lib/agent-plan" -import type { AgentType, ConnectionStatus, MessageTurn } from "@/lib/types" +import type { + AgentType, + ConnectionStatus, + MessageTurn, + SearchMatchLocation, +} from "@/lib/types" import { copyTextToClipboard } from "@/lib/utils" import { VirtualizedMessageThread } from "@/components/message/virtualized-message-thread" import { @@ -105,6 +110,12 @@ interface MessageListViewProps { * items render in arbitrary order and multiplicity. `null` = no divider. */ userTurnHeader?: ((group: ResolvedMessageGroup) => string | null) | null + searchMatch?: (SearchMatchLocation & { occurrenceIndex?: number }) | null + searchQuery?: string | null + searchMatchOrdinal?: number | null + searchMatchTotal?: number | null + onNextMatch?: () => void + onSearchNavigationFailed?: () => void } export interface ResolvedMessageGroup { @@ -661,6 +672,240 @@ const AutoScrollOnSend = memo(function AutoScrollOnSend({ return null }) +const SEARCH_HIGHLIGHT_KEY = "codeg-search-hit" + +function findSearchTurnElement( + root: HTMLElement | null, + turnId: string +): HTMLElement | null { + if (!root) return null + const candidates = root.querySelectorAll( + "[data-search-turn-ids]" + ) + for (const element of Array.from(candidates).reverse()) { + try { + const ids = JSON.parse(element.dataset.searchTurnIds ?? "[]") as string[] + if (ids.includes(turnId)) return element + } catch { + // Ignore malformed attributes. + } + } + return null +} + +function findScrollableAncestor( + element: HTMLElement | null +): HTMLElement | null { + let current: HTMLElement | null = element + while (current) { + if (current.scrollHeight > current.clientHeight + 1) { + const overflowY = window.getComputedStyle(current).overflowY + if (overflowY === "auto" || overflowY === "scroll") return current + } + current = current.parentElement + } + return null +} + +function scrollElementToCenter(element: HTMLElement) { + const container = findScrollableAncestor(element) + if (!container) return + const targetRect = element.getBoundingClientRect() + const containerRect = container.getBoundingClientRect() + const nextScrollTop = + container.scrollTop + + targetRect.top - + containerRect.top - + (container.clientHeight - element.clientHeight) / 2 + container.scrollTo({ + top: Math.max(0, nextScrollTop), + behavior: "smooth", + }) +} + +function scrollRangeToCenter(range: Range) { + const node = range.startContainer + const element = + node instanceof HTMLElement ? node : (node.parentElement ?? null) + const container = findScrollableAncestor(element) + if (!container) return + const targetRect = range.getBoundingClientRect() + const containerRect = container.getBoundingClientRect() + const nextScrollTop = + container.scrollTop + + targetRect.top - + containerRect.top - + (container.clientHeight - targetRect.height) / 2 + container.scrollTo({ + top: Math.max(0, nextScrollTop), + behavior: "smooth", + }) +} + +/** The exact matched substring from the source block, taken from the backend + * char offsets. Using this per-match text (rather than the whole query) + * keeps multi-term highlights and their counts aligned with the backend. */ +function searchMatchNeedle( + match: SearchMatchLocation, + turn: MessageTurn | undefined +): string | null { + if (!turn || match.block_index == null) return null + const block = turn.blocks[match.block_index] + if (block?.type !== "text") return null + const chars = Array.from(block.text) + const start = Math.max(0, Math.min(match.char_start, chars.length)) + const end = Math.max(start, Math.min(match.char_end, chars.length)) + if (start >= end) return null + return chars.slice(start, end).join("") +} + +function countOccurrences(text: string, needleLower: string): number { + if (!needleLower) return 0 + const lower = text.toLocaleLowerCase() + let count = 0 + let from = 0 + while (from <= lower.length - needleLower.length) { + const index = lower.indexOf(needleLower, from) + if (index < 0) break + count += 1 + from = index + needleLower.length + } + return count +} + +/** How many occurrences of `needle` appear in the rendered DOM before this + * match. Merged assistant turns share one DOM element, so the raw per-turn + * occurrence index alone would land on the wrong turn; summing the source + * turns that render before the target block fixes that offset. */ +function searchOccurrenceOffset( + threadItems: ThreadRenderItem[], + targetIndex: number, + turnId: string, + blockIndex: number, + charStart: number, + needleLower: string +): number { + let offset = 0 + for (let index = 0; index < targetIndex; index += 1) { + const item = threadItems[index] + if (item.kind !== "turn") continue + for (const turn of item.sourceTurns) { + for (const block of turn.blocks) { + if (block.type === "text") + offset += countOccurrences(block.text, needleLower) + } + } + } + const target = threadItems[targetIndex] + if (!target || target.kind !== "turn") return offset + for (const turn of target.sourceTurns) { + if (turn.id !== turnId) { + for (const block of turn.blocks) { + if (block.type === "text") + offset += countOccurrences(block.text, needleLower) + } + continue + } + for ( + let index = 0; + index < Math.min(blockIndex, turn.blocks.length); + index += 1 + ) { + const block = turn.blocks[index] + if (block.type === "text") + offset += countOccurrences(block.text, needleLower) + } + const block = turn.blocks[blockIndex] + if (block?.type === "text") { + const chars = Array.from(block.text) + const prefix = chars.slice(0, Math.min(charStart, chars.length)).join("") + offset += countOccurrences(prefix, needleLower) + } + break + } + return offset +} + +function createHighlightRange( + element: HTMLElement, + needle: string, + occurrence: number +): Range | null { + const needleLower = needle.toLocaleLowerCase() + if (!needleLower) return null + const walker = document.createTreeWalker(element, NodeFilter.SHOW_TEXT) + let node = walker.nextNode() + let count = 0 + while (node) { + const parentTag = node.parentElement?.tagName ?? "" + if (!["SCRIPT", "STYLE", "NOSCRIPT"].includes(parentTag)) { + const text = node.nodeValue ?? "" + const lower = text.toLocaleLowerCase() + let offset = 0 + while (offset <= lower.length - needleLower.length) { + const index = lower.indexOf(needleLower, offset) + if (index < 0) break + if (count === occurrence) { + const range = document.createRange() + range.setStart(node, index) + range.setEnd(node, index + needleLower.length) + return range + } + count += 1 + offset = index + needleLower.length + } + } + node = walker.nextNode() + } + return null +} + +interface SearchHighlightRegistry { + set(name: string, highlight: unknown): void + delete(name: string): void +} + +let searchHighlightStyleInjected = false + +/** The static CSS pipeline does not accept `::highlight()` yet, so inject the + * one rule at runtime and reuse the theme's yellow token. */ +function ensureSearchHighlightStyle() { + if (searchHighlightStyleInjected) return + searchHighlightStyleInjected = true + const style = document.createElement("style") + style.id = "codeg-search-highlight-style" + const background = + getComputedStyle(document.documentElement) + .getPropertyValue("--search-hit-bg") + .trim() || "#fde047" + style.textContent = `::highlight(codeg-search-hit){background-color:${background};color:inherit}` + document.head.appendChild(style) +} + +/** Apply a highlight without mutating React-owned DOM. Falls back to nothing + * (scroll only) on engines that predate the CSS Custom Highlight API. */ +function applySearchHighlight(range: Range): (() => void) | null { + if (typeof CSS === "undefined") return null + const registry = (CSS as unknown as { highlights?: SearchHighlightRegistry }) + .highlights + const HighlightCtor = ( + window as unknown as { + Highlight?: new (...ranges: Range[]) => { add: (range: Range) => void } + } + ).Highlight + if (!registry || !HighlightCtor) return null + ensureSearchHighlightStyle() + const highlight = new HighlightCtor() + highlight.add(range) + registry.set(SEARCH_HIGHLIGHT_KEY, highlight) + const timer = window.setTimeout(() => { + registry.delete(SEARCH_HIGHLIGHT_KEY) + }, 1800) + return () => { + window.clearTimeout(timer) + registry.delete(SEARCH_HIGHLIGHT_KEY) + } +} export function MessageListView({ conversationId, agentType, @@ -675,6 +920,12 @@ export function MessageListView({ onNewSession, showMessageNav = true, userTurnHeader = null, + searchMatch = null, + searchQuery = null, + searchMatchOrdinal = null, + searchMatchTotal = null, + onNextMatch, + onSearchNavigationFailed, }: MessageListViewProps) { const t = useTranslations("Folder.chat.messageList") const sharedT = useTranslations("Folder.chat.shared") @@ -890,7 +1141,12 @@ export function MessageListView({ ? userTurnHeader(item.group) : null return ( -
0 ? { paddingTop: pt } : undefined}> +
turn.id) + )} + style={pt > 0 ? { paddingTop: pt } : undefined} + > {phaseLabel ? (