Retire the in-process BM25 text stack; tests and evals measure the store-native FTS

BENCHMARK.md

@@ -222,12 +222,21 @@ versus an LLM-judged setup.
 
 Per-tier R@5 (bm25): exact **96.8%** · concept 25.4% · multi_hop 30.0%.
 
+_The `bm25` row key is historical. The row measures the shipping
+lexical path — the store's native symbol FTS queried through
+`Engine.SearchSymbols`, the same call `search_symbols` makes — not an
+in-process BM25 index. The key is kept unchanged so older runs stay
+joinable._
+
 **Headline**: the `search_symbols` text path (`bm25`) lands
 **R@5 = 55.1%** / **R@20 = 63.5%**, and **96.8%** on exact
 symbol-name queries — 3.2× ripgrep's R@5 floor. Enabling Porter
 stemming (`GORTEX_FTS_STEMMING=1`) trades a little exact-tier
 precision for breadth — R@20 +5.7pp, exact-tier R@5 −3.1pp — so it
-ships opt-in. The `semantic` and `rrf` rankers require `--embeddings`
+ships opt-in. The mode is read at process start; on a later restart,
+gortex automatically rebuilds a repository's native symbol FTS when
+its persisted normalization marker differs. The `semantic` and `rrf`
+rankers require `--embeddings`
 and are omitted here; the `graph` ranker scores only graph-traversal
 fixtures.
 

CONTRIBUTING.md

@@ -236,7 +236,7 @@ internal/
   query/             Query engine (BFS traversal, SubGraph)
   resolver/          Cross-file reference resolution, IMPLEMENTS inference
   review/            PR review and diff analysis pipeline
-  search/            BM25, trigram, and semantic search
+  search/            FTS adapter, trigram, and semantic search
   server/            HTTP transports and the web dashboard
 ```
 

README.md

@@ -50,7 +50,7 @@ High-quality parsing 257 languages/grammars through tree-sitter AST analysis, in
 - **Zero external dependencies** — single binary, everything in-process. No network, no model download to get started. Install, start daemon, use.
 - **Agent integrations (19)** — `gortex init` configures every detected coding assistant on the machine → [docs/agents.md](docs/agents.md)
 - **100+ MCP tools, 16 resources, 3 prompts** — symbol lookup, call chains, blast radius, dataflow, clone detection, refactoring, code actions → [docs/mcp.md](docs/mcp.md)
-- **Semantic search default-on** — baked GloVe-50d (3.8 MB embedded), hybrid BM25 + vector + RRF, zero deps; opt-in MiniLM / Ollama / OpenAI → [docs/semantic-search.md](docs/semantic-search.md)
+- **Semantic search default-on** — baked GloVe-50d (3.8 MB embedded), store-native FTS5/BM25 + vector with adaptive alpha fusion, zero deps; opt-in MiniLM / Ollama / OpenAI → [docs/semantic-search.md](docs/semantic-search.md)
 - **Speculative execution** — `preview_edit` / `simulate_chain` answer "what would change if I applied this WorkspaceEdit?" without touching disk
 - **Live editor overlays** — push unsaved buffers as a shadow graph; tools read through it. Branching for parallel speculative sessions
 - **GCX1 wire format** — published, round-trippable. **An additional −27% tokens vs JSON** at same fidelity → [docs/wire-format.md](docs/wire-format.md)

bench/fixtures/retrieval.yaml

@@ -27,11 +27,11 @@
 # Cases are tiered so per-tier weakness is visible:
 #
 #   tier: exact      — symbol-name queries. Tests "find this named
-#                      thing." BM25 should dominate; a retrieval stack
-#                      that can't ace exact tier is broken.
+#                      thing." Store-native lexical search should dominate;
+#                      a retrieval stack that can't ace exact tier is broken.
 #   tier: concept    — natural-language paraphrase queries. Tests
-#                      semantic understanding. Where BM25 starts losing
-#                      to semantic / RRF if the embedder is competent.
+#                      semantic understanding. Where lexical retrieval starts
+#                      losing to vector/adaptive fusion if the embedder is competent.
 #   tier: multi_hop  — relational queries with several valid expected
 #                      IDs (any-hit semantics). Tests graph-aware
 #                      retrieval: "things that handle X" / "extractors
@@ -63,8 +63,8 @@ cases:
   - { id: exact-AddNode,             tier: exact, query: "AddNode",                     expected: [internal/graph/graph.go::Graph.AddNode] }
   - { id: exact-AtomicWriteFile,     tier: exact, query: "AtomicWriteFile",             expected: [internal/agents/writer.go::AtomicWriteFile] }
   - { id: exact-WriteIfNotExists,    tier: exact, query: "WriteIfNotExists",            expected: [internal/agents/writer.go::WriteIfNotExists] }
-  - { id: exact-BM25Backend,         tier: exact, query: "BM25Backend",                 expected: [internal/search/bm25.go::BM25Backend] }
-  - { id: exact-NewBM25,             tier: exact, query: "NewBM25",                     expected: [internal/search/bm25.go::NewBM25] }
+  - { id: exact-SymbolSearcherBackend, tier: exact, query: "SymbolSearcherBackend",       expected: [internal/search/symbolsearcher_backend.go::SymbolSearcherBackend] }
+  - { id: exact-NewSymbolSearcherBackend, tier: exact, query: "NewSymbolSearcherBackend", expected: [internal/search/symbolsearcher_backend.go::NewSymbolSearcherBackend] }
   - { id: exact-HybridBackend,       tier: exact, query: "HybridBackend",               expected: [internal/search/hybrid.go::HybridBackend] }
   - { id: exact-NewHybrid,           tier: exact, query: "NewHybrid",                   expected: [internal/search/hybrid.go::NewHybrid] }
   - { id: exact-alphaFuse,           tier: exact, query: "alphaFuse",                   expected: [internal/search/hybrid.go::alphaFuse] }
@@ -119,10 +119,10 @@ cases:
   - { id: concept-reindex-one-file,  tier: concept, query: "reindex a single file after an edit",           expected: [internal/indexer/indexer.go::Indexer.IndexFile] }
   - { id: concept-evict-file,        tier: concept, query: "remove all nodes belonging to a file",          expected: [internal/graph/graph.go::Graph.EvictFile] }
   - { id: concept-evict-repo,        tier: concept, query: "drop every node for a repository prefix",       expected: [internal/graph/graph.go::Graph.EvictRepo] }
-  - { id: concept-bm25-index,        tier: concept, query: "text search backend with TF-IDF ranking",       expected: [internal/search/bm25.go::BM25Backend] }
-  - { id: concept-hybrid-fuse,       tier: concept, query: "combine text and vector search with RRF",       expected: [internal/search/hybrid.go::HybridBackend] }
-  - { id: concept-adaptive-alpha,    tier: concept, query: "adaptive alpha weighted reciprocal rank fusion", expected: [internal/search/hybrid.go::alphaFuse] }
-  - { id: concept-swap-backend,      tier: concept, query: "hot-swap the in-memory search backend",         expected: [internal/search/swappable.go::Swappable] }
+  - { id: concept-store-native-text, tier: concept, query: "adapt store-native symbol search into the search backend", expected: [internal/search/symbolsearcher_backend.go::SymbolSearcherBackend] }
+  - { id: concept-hybrid-fuse,       tier: concept, query: "combine store-native text and vector search with adaptive alpha", expected: [internal/search/hybrid.go::HybridBackend] }
+  - { id: concept-adaptive-alpha,    tier: concept, query: "adaptive alpha weighted text and vector fusion", expected: [internal/search/hybrid.go::alphaFuse] }
+  - { id: concept-swap-backend,      tier: concept, query: "atomically replace the vector channel while preserving the live text backend", expected: [internal/search/swappable.go::Swappable.ReplaceHybridVector] }
   - { id: concept-glove-embed,       tier: concept, query: "built-in GloVe word vector embedder",           expected: [internal/embedding/static.go::StaticProvider] }
   - { id: concept-mcp-server,        tier: concept, query: "MCP server type holding engine and graph",      expected: [internal/mcp/server.go::Server] }
   - { id: concept-session-state,     tier: concept, query: "per-client session activity tracking",          expected: [internal/mcp/server.go::sessionState] }
@@ -154,7 +154,7 @@ cases:
   - { id: concept-ts-extract,        tier: concept, query: "TypeScript language extractor",                 expected: [internal/parser/languages/typescript.go::TypeScriptExtractor] }
   - { id: concept-detect-lang,       tier: concept, query: "detect language from file extension",           expected: [internal/parser/registry.go::Registry.GetByExtension] }
   - { id: concept-register-all,      tier: concept, query: "register every language extractor",             expected: [internal/parser/languages/register.go::RegisterAll] }
-  - { id: concept-ranker-bm25,       tier: concept, query: "eval ranker adapter for BM25",                  expected: [internal/eval/recall/rankers.go::BM25Ranker] }
+  - { id: concept-ranker-engine,     tier: concept, query: "eval ranker adapter for engine search",         expected: [internal/eval/recall/rankers.go::EngineRanker] }
   - { id: concept-ranker-rrf,        tier: concept, query: "eval ranker adapter for RRF hybrid",            expected: [internal/eval/recall/rankers.go::RRFRanker] }
   - { id: concept-ranker-semantic,   tier: concept, query: "eval ranker for vector-only semantic",          expected: [internal/eval/recall/rankers.go::SemanticRanker] }
   - { id: concept-ranker-winnow,     tier: concept, query: "eval ranker wrapping graph-aware winnow",       expected: [internal/eval/recall/rankers.go::WinnowRanker] }
@@ -163,7 +163,7 @@ cases:
   - { id: concept-repo-for-file,     tier: concept, query: "find which repo contains a given file",         expected: [internal/indexer/multi.go::MultiIndexer.RepoForFile] }
   - { id: concept-set-embedder,      tier: concept, query: "attach embedding provider to the indexer",      expected: [internal/indexer/indexer.go::Indexer.SetEmbedder] }
   - { id: concept-new-server,        tier: concept, query: "construct a new MCP server instance",           expected: [internal/mcp/server.go::NewServer] }
-  - { id: concept-new-hybrid,        tier: concept, query: "construct a hybrid BM25+vector backend",        expected: [internal/search/hybrid.go::NewHybrid] }
+  - { id: concept-new-hybrid,        tier: concept, query: "construct a hybrid store-native text and vector backend", expected: [internal/search/hybrid.go::NewHybrid] }
   - { id: concept-reindex-all,       tier: concept, query: "incremental reindex after file changes",        expected: [internal/indexer/indexer.go::Indexer.IncrementalReindexPaths] }
   - { id: concept-new-vector,        tier: concept, query: "HNSW vector index backend",                     expected: [internal/search/vector.go::VectorBackend] }
   - { id: concept-api-embedder,      tier: concept, query: "OpenAI-compatible embeddings API provider",     expected: [internal/embedding/api.go::APIProvider] }
@@ -229,7 +229,7 @@ cases:
     tier: multi_hop
     query: "search backend implementations"
     expected:
-      - internal/search/bm25.go::BM25Backend
+      - internal/search/null.go::NullBackend
       - internal/search/symbolsearcher_backend.go::SymbolSearcherBackend
       - internal/search/hybrid.go::HybridBackend
       - internal/search/vector.go::VectorBackend
@@ -269,7 +269,7 @@ cases:
     tier: multi_hop
     query: "ranker adapters in the eval package"
     expected:
-      - internal/eval/recall/rankers.go::BM25Ranker
+      - internal/eval/recall/rankers.go::EngineRanker
       - internal/eval/recall/rankers.go::SemanticRanker
       - internal/eval/recall/rankers.go::RRFRanker
       - internal/eval/recall/rankers.go::WinnowRanker
@@ -410,15 +410,15 @@ cases:
       - internal/config/config.go::Load
       - internal/config/config.go::Default
 
-  - id: mh-bm25-api
+  - id: mh-null-backend-api
     tier: multi_hop
-    query: "BM25 backend public API"
+    query: "null search backend public API"
     expected:
-      - internal/search/bm25.go::BM25Backend.Add
-      - internal/search/bm25.go::BM25Backend.Remove
-      - internal/search/bm25.go::BM25Backend.Search
-      - internal/search/bm25.go::BM25Backend.Count
-      - internal/search/bm25.go::NewBM25
+      - internal/search/null.go::NullBackend.Add
+      - internal/search/null.go::NullBackend.Remove
+      - internal/search/null.go::NullBackend.Search
+      - internal/search/null.go::NullBackend.Count
+      - internal/search/null.go::NewNull
 
   - id: mh-tool-registrations
     tier: multi_hop

bench/fixtures/retrieval_typo.yaml

@@ -45,79 +45,79 @@ cases:
       query: IndexFileNozResolve
       expected:
         - internal/indexer/indexer.go::Indexer.IndexFileNoResolve
-    - id: exact-IncrementalReindex-typo
+    - id: exact-IncrementalReindexPaths-typo
       tier: exact
-      query: IncrementalPeindex
+      query: IncrementalReigdexPaths
       expected:
         - internal/indexer/indexer.go::Indexer.IncrementalReindexPaths
     - id: exact-SetEmbedder-typo
       tier: exact
-      query: SetEmbeeder
+      query: SetEmbedde
       expected:
         - internal/indexer/indexer.go::Indexer.SetEmbedder
     - id: exact-EvictFile-typo
       tier: exact
-      query: EvictFife
+      query: EvictFilq
       expected:
         - internal/graph/graph.go::Graph.EvictFile
     - id: exact-EvictRepo-typo
       tier: exact
-      query: EvkctRepo
+      query: EiictRepo
       expected:
         - internal/graph/graph.go::Graph.EvictRepo
     - id: exact-AddNode-typo
       tier: exact
-      query: AddNode
+      query: AddoNde
       expected:
         - internal/graph/graph.go::Graph.AddNode
     - id: exact-AtomicWriteFile-typo
       tier: exact
-      query: AtomicnWriteFile
+      query: AtomicWriteile
       expected:
         - internal/agents/writer.go::AtomicWriteFile
     - id: exact-WriteIfNotExists-typo
       tier: exact
-      query: WrilteIfNotExists
+      query: WriteIfNotExisks
       expected:
         - internal/agents/writer.go::WriteIfNotExists
-    - id: exact-BM25Backend-typo
+    - id: exact-SymbolSearcherBackend-typo
       tier: exact
-      query: BM25Backejd
+      query: SymbolSearcherackend
       expected:
-        - internal/search/bm25.go::BM25Backend
-    - id: exact-NewBM25-typo
+        - internal/search/symbolsearcher_backend.go::SymbolSearcherBackend
+    - id: exact-NewSymbolSearcherBackend-typo
       tier: exact
-      query: NenBM25
+      query: NewSymboSlearcherBackend
       expected:
-        - internal/search/bm25.go::NewBM25
+        - internal/search/symbolsearcher_backend.go::NewSymbolSearcherBackend
     - id: exact-HybridBackend-typo
       tier: exact
-      query: HybridBfckend
+      query: HybridUackend
       expected:
         - internal/search/hybrid.go::HybridBackend
     - id: exact-NewHybrid-typo
       tier: exact
-      query: NewHybid
+      query: NtwHybrid
       expected:
         - internal/search/hybrid.go::NewHybrid
     - id: exact-alphaFuse-typo
       tier: exact
-      query: alphaFse
+      query: agphaFuse
       expected:
         - internal/search/hybrid.go::alphaFuse
     - id: exact-VectorBackend-typo
       tier: exact
-      query: VectorBcakend
+      query: VectorBackedd
       expected:
         - internal/search/vector.go::VectorBackend
     - id: exact-SearchBackend-typo
       tier: exact
-      query: search.Backeod
+      query: sewarch.Backend
       expected:
         - internal/search/search.go::Backend
     - id: exact-SearchResult-typo
       tier: exact
-      query: SearchResutl
+      query: SearchResuzt
       expected:
         - internal/search/search.go::SearchResult
     - id: exact-Swappable-typo
@@ -335,26 +335,26 @@ cases:
       query: drop every node for a rpeository prefix
       expected:
         - internal/graph/graph.go::Graph.EvictRepo
-    - id: concept-bm25-index-typo
+    - id: concept-store-native-text-typo
       tier: concept
-      query: text search backend wtih TF-IDF ranking
+      query: adapt store-native symbol search itno the search backend
       expected:
-        - internal/search/bm25.go::BM25Backend
+        - internal/search/symbolsearcher_backend.go::SymbolSearcherBackend
     - id: concept-hybrid-fuse-typo
       tier: concept
-      query: comlbine text and vector search with RRF
+      query: combine store-native text and lvector search with adaptive alpha
       expected:
         - internal/search/hybrid.go::HybridBackend
     - id: concept-adaptive-alpha-typo
       tier: concept
-      query: adaptive alpha weighted reciprocal randk fusion
+      query: adaptive alpha weightded text and vector fusion
       expected:
         - internal/search/hybrid.go::alphaFuse
     - id: concept-swap-backend-typo
       tier: concept
-      query: hot-swap the in-memory searich backend
+      query: atomically replace the vectior channel while preserving the live text backend
       expected:
-        - internal/search/swappable.go::Swappable
+        - internal/search/swappable.go::Swappable.ReplaceHybridVector
     - id: concept-glove-embed-typo
       tier: concept
       query: built-in GloVe owrd vector embedder
@@ -510,11 +510,11 @@ cases:
       query: register every language extactor
       expected:
         - internal/parser/languages/register.go::RegisterAll
-    - id: concept-ranker-bm25-typo
+    - id: concept-ranker-engine-typo
       tier: concept
-      query: eval rnker adapter for BM25
+      query: eval rnker adapter for engine search
       expected:
-        - internal/eval/recall/rankers.go::BM25Ranker
+        - internal/eval/recall/rankers.go::EngineRanker
     - id: concept-ranker-rrf-typo
       tier: concept
       query: eval ranker dapter for RRF hybrid
@@ -557,7 +557,7 @@ cases:
         - internal/mcp/server.go::NewServer
     - id: concept-new-hybrid-typo
       tier: concept
-      query: construct a hbyrid BM25+vector backend
+      query: construct a hybrid stroe-native text and vector backend
       expected:
         - internal/search/hybrid.go::NewHybrid
     - id: concept-reindex-all-typo
@@ -672,7 +672,7 @@ cases:
       tier: multi_hop
       query: search backend implementatsions
       expected:
-        - internal/search/bm25.go::BM25Backend
+        - internal/search/null.go::NullBackend
         - internal/search/symbolsearcher_backend.go::SymbolSearcherBackend
         - internal/search/hybrid.go::HybridBackend
         - internal/search/vector.go::VectorBackend
@@ -708,7 +708,7 @@ cases:
       tier: multi_hop
       query: ranker adapters in the eval packapge
       expected:
-        - internal/eval/recall/rankers.go::BM25Ranker
+        - internal/eval/recall/rankers.go::EngineRanker
         - internal/eval/recall/rankers.go::SemanticRanker
         - internal/eval/recall/rankers.go::RRFRanker
         - internal/eval/recall/rankers.go::WinnowRanker
@@ -832,15 +832,15 @@ cases:
       expected:
         - internal/config/config.go::Load
         - internal/config/config.go::Default
-    - id: mh-bm25-api-typo
+    - id: mh-null-backend-api-typo
       tier: multi_hop
-      query: BM25 backend pubilc API
+      query: null seacrh backend public API
       expected:
-        - internal/search/bm25.go::BM25Backend.Add
-        - internal/search/bm25.go::BM25Backend.Remove
-        - internal/search/bm25.go::BM25Backend.Search
-        - internal/search/bm25.go::BM25Backend.Count
-        - internal/search/bm25.go::NewBM25
+        - internal/search/null.go::NullBackend.Add
+        - internal/search/null.go::NullBackend.Remove
+        - internal/search/null.go::NullBackend.Search
+        - internal/search/null.go::NullBackend.Count
+        - internal/search/null.go::NewNull
     - id: mh-tool-registrations-typo
       tier: multi_hop
       query: MCP tool registration functuons