[BUG] BM25 search fails: position is not found but required for phrase queries

[jlin53882]

🐛 Bug Report: BM25 search fails with "position is not found but required for phrase queries"

Environment

• Plugin version: memory-lancedb-pro@1.1.0-beta.10
• LanceDB version: 0.26.2 (bundled with plugin)
• OpenClaw version: 2026.03.x (latest)
• Platform: Windows_11 (observed on Gateway)

---

Error

Every BM25 query fails with this error, causing a fallback to empty results:

BM25 search failed, falling back to empty results: [Error: Failed to execute query stream: GenericFailure, lance error: Invalid user input: position is not found but required for phrase queries, try recreating the index with position, ...]
Caused by: position is not found but required for phrase queries, try recreating the index with position

Root Cause

In src/store.ts, the createFtsIndex() method creates the FTS index without position data:

await table.createIndex("text", {
  config: (lancedb as any).Index.fts(),  // withPosition defaults to false
});

Index.fts() in LanceDB 0.26.2 defaults withPosition: false. When a phrase query is executed internally (e.g., multi-word search treated as phrase), LanceDB requires position data that does not exist in the index, causing the error.

The native.d.ts signature confirms:

static fts(
  withPosition?: boolean,  // ← first arg, defaults to false
  baseTokenizer?: string,
  language?: string,
  ...
): Index

Impact

• All BM25/keyword searches silently fall back to empty results
• Users lose full-text search capability entirely
• Error is caught and suppressed (console.warn), so users don't see the actual error — they just get no results

Workaround

The plugin already has rebuildFtsIndex() method in 1.1.0-beta.10 which can recreate the index. However:

1. Users need to know to call it manually
2. The createFtsIndex() source code still uses the wrong default

Suggested Fix

Option A — Fix at creation time (preferred, prevents future occurrences):

// src/store.ts, createFtsIndex()
config: (lancedb as any).Index.fts(true),  // explicitly enable position data

Option B — Add rebuildFtsIndex to auto-trigger when FTS query fails with this specific error, instead of just falling back silently.

---

Additional Notes

• The plugin already has a rebuildFtsIndex() public method that can fix existing deployments
• This error started appearing after upgrading to beta.10 / LanceDB 0.26.2 (previously on 1.0.32 with earlier LanceDB this did not occur)
• The error is 100% reproducible on any BM25 query once the index lacks position data

[jlin53882]

驗證結果（已實際重現）

已在本地環境（memory-lancedb-pro@1.1.0-beta.10 + LanceDB 0.26.2）完成驗證，100% 重現確認。

實驗環境

• DB Path: ~/.openclaw/memory/lancedb-pro-jina1024
• Plugin 版本: memory-lancedb-pro@1.1.0-beta.10
• LanceDB 版本: 0.26.2

實驗結果

測試	查詢類型	結果
Test 1	單一關鍵字 hello	OK (2 results)
Test 2	短語查詢 "hello world"	FAIL - position is not found but required for phrase queries
Test 3	多詞查詢 hello world	OK (2 results)
Test 4	Index.fts() 無參數	確認 withPosition 預設為 false

現有 Index 狀態

name: "text_idx"
indexType: "FTS"
columns: ["text"]

index 存在但無 position 資料。

根因鏈

store.ts -> createFtsIndex() -> Index.fts()     <- withPosition 預設 false
                                            |
                                  短語查詢需要 position
                                            |
                              phrase query -> ERROR

觸發條件

• 單詞查詢：不需要 position，正常
• 短語查詢（雙引號 "..."）：需要 position，觸發錯誤

已成功用 Node.js 直接對實際資料庫執行驗證，錯誤訊息與日誌完全一致。

修復建議（已確認有效）: Index.fts(true) 可解決問題。

[jlin53882]

更正：修復方案需要修正（已驗證）

經過 30 次迭代驗證（150 個測試 case），發現：

⚠️ 原始修復方案有誤

Index.fts() 的 API 不是 positional arguments，而是 options object：

// indices.js - 實際 API 定義
static fts(options) {  // ← 參數是 object，不是 boolean
    return new Index(native_1.Index.fts(
        options?.withPosition,   // options 是 { withPosition: true } 才有值
        ...
    ));
}

所以 Index.fts(true) 在 JavaScript 中：

Index.fts(true)  // → options = true (boolean)
options?.withPosition  // → undefined（boolean 沒有 withPosition 屬性）

✅ 正確修復方案

// 錯誤（無效）
config: (lancedb as any).Index.fts(true),           // ← withPosition = undefined

// 正確
config: (lancedb as any).Index.fts({ withPosition: true }),  // ← withPosition = true

驗證結果

在 test-pr354 隔離環境中完成：

• 錯誤 API Index.fts(true)：phrase query 失敗 ✅（確認是 bug）
• 正確 API Index.fts({ withPosition: true })：
  • 30 次迭代
  • 每次 5 個測試 case
  • 總計 150 個測試
  • 150 PASS / 0 FAIL
  • 成功率 100%

受影響檔案

• src/store.ts 的 createFtsIndex() 方法
• dist/src/store.js 編譯後版本

觸發條件（已驗證）

查詢類型	Index.fts() 無參數	Index.fts({ withPosition: true })
單詞 hello	✅ PASS	✅ PASS
短語 "hello world"	❌ FAIL	✅ PASS
多詞 hello world	✅ PASS	✅ PASS