Push filtering DB scans into SQL on the auto-index path

[voxmenthe]

Summary

colgrep runs try_index on every search. Two helpers on that path were materializing every metadata row of the entire index just to inspect a single column:

• cleanup_orphaned_entries (colgrep/src/index/mod.rs) called filtering::get(index_path, None, &[], None) solely to collect the distinct file paths from the result. That executes SELECT * FROM METADATA ORDER BY _subset_ and JSON-deserializes every column — including code — for every row.
• reconcile_document_counts (colgrep/src/index/mod.rs) did the same full-table dump, then iterated in Rust to find rows whose _subset_ exceeded the vector index document count.

On a ~95 k-unit index this is tens of MB of deserialization on every query, which dominates latency for otherwise-quick searches and is a frequent cause of "colgrep hangs after the desync warning" reports.

What changed

• New helper filtering::get_distinct_strings(index_path, column) in next-plaid/src/filtering.rs. Runs a single SELECT DISTINCT "<col>" FROM METADATA WHERE "<col>" IS NOT NULL, validates the column name as a safe identifier, and returns it as Vec<String>. Returns an empty vector when the DB or column is absent (matching the previous lenient behavior).
• cleanup_orphaned_entries now uses get_distinct_strings(index_path, "file") instead of loading every row.
• reconcile_document_counts now uses the existing filtering::where_condition(index_path, "_subset_ >= ?", &[json!(vector_count)]) to fetch orphan IDs directly. The reconciliation semantics are unchanged.

Both reads now transfer a few KB instead of the whole METADATA table.

Performance

Reproducible benchmark added at next-plaid/examples/bench_metadata_filter.rs. It builds a synthetic METADATA table at varying sizes and times the OLD path (full-table filtering::get + Rust-side filter, exactly what the two functions were doing) against the NEW SQL pushdown. Best-of-5, 2024 MacBook release build, 600 B of synthetic code per row, 8 units per file:

n_rows	op	OLD time	NEW time	speedup	bytes (OLD → NEW)
10 k	distinct files (cleanup)	35.7 ms	3.2 ms	11.2×	10.21 MB → 33.93 KB
10 k	orphan ids (reconcile)	34.7 ms	664.3 µs	52.2×	10.21 MB → 6 B
50 k	distinct files (cleanup)	181.7 ms	17.0 ms	10.7×	51.29 MB → 174.00 KB
50 k	orphan ids (reconcile)	172.7 ms	542.4 µs	318.3×	51.29 MB → 31 B
95 k	distinct files (cleanup)	341.8 ms	32.0 ms	10.7×	97.53 MB → 333.43 KB
95 k	orphan ids (reconcile)	330.5 ms	586.0 µs	564.0×	97.53 MB → 61 B
200 k	distinct files (cleanup)	718.9 ms	73.2 ms	9.8×	205.94 MB → 713.94 KB
200 k	orphan ids (reconcile)	696.6 ms	592.0 µs	1176.8×	205.94 MB → 141 B

The bench asserts result-set equivalence between OLD and NEW before timing, so it doubles as a regression check. Reproduce with:

cargo run --release --example bench_metadata_filter -p next-plaid -- --quick
cargo run --release --example bench_metadata_filter -p next-plaid -- \
    --sizes 10000,50000,95000,200000 --iters 5

cleanup_orphaned_entries runs on every search, so its 300 ms+ savings (at 95 k rows) hit every query. reconcile_document_counts runs only when the index/DB counts disagree, but that is exactly the state where users observe minute-long stalls.

Tests

• New unit tests in next-plaid/src/filtering.rs:
  • test_get_distinct_strings_returns_unique_values
  • test_get_distinct_strings_missing_db_returns_empty
  • test_get_distinct_strings_unknown_column_returns_empty
  • test_get_distinct_strings_rejects_invalid_column_name
• cargo test -p next-plaid --lib filtering:: — 44 passed (4 new, 40 existing)
• cargo test -p colgrep --lib — 511 passed
• cargo check -p next-plaid -p colgrep — clean

The benchmark example is in a separate commit so it can be dropped cleanly if you'd rather not add example binaries to the crate.

Test plan

• New helper covered by unit tests (happy path, missing DB, unknown column, invalid identifier)
• Existing filtering test suite still passes
• Full colgrep lib test suite still passes
• cargo check passes for both crates
• Bench asserts OLD/NEW result equivalence before timing

[raphaelsty]

Hi @voxmenthe this is a cool PR, I'd like simply to see a benchmark please to merge, how much time it save / memory

[voxmenthe]

Hi @voxmenthe this is a cool PR, I'd like simply to see a benchmark please to merge, how much time it save / memory

Added to PR thanks!