Add benchmark-driven performance optimizations

[Shengyu Fu (shengyfu)]

Summary

• add Criterion benchmarks for trigram extraction, query execution, and index builds, including on-disk query and high-diversity index workloads
• optimize OR query unions, lowercase trigram extraction, index-build memory usage, and index serialization
• replace full-build inverted HashMap<u32, Vec<PostingEntry>> construction with flat postings sorted/grouped for writing
• skip redundant query-time posting sort/dedup when on-disk posting lists are already sorted
• bump workspace version to 0.1.20

Benchmark highlights

• high-diversity 1,000-file index build: ~1.4659s -> 354.73ms, peak working set ~98.74MiB -> 43.58MiB
• on-disk common literal query, 5,000 files: ~645.85us -> 336.27us
• OR union query execution: previously measured ~70%+ faster across larger branch counts

Validation

• cargo check --workspace
• cargo fmt --all -- --check
• cargo test --workspace
• cargo clippy --workspace --benches -- -D warnings

[Copilot]

Pull request overview

This PR adds a benchmark suite around tgrep-core and applies several performance-oriented changes to trigram extraction, query execution, and index serialization. It mainly targets the core indexing/query pipeline, where tgrep spends most of its time when building indexes and evaluating searches.

Changes:

• Add Criterion benchmarks for trigram extraction, query execution, and index builds.
• Optimize OR-query execution, lowercase trigram extraction, and on-disk index writing to reduce work and memory usage.
• Bump the workspace/package version to 0.1.20 and update the lockfile for new benchmark dependencies.

Show a summary per file

File	Description
tgrep-core/src/trigram.rs	Optimizes merged trigram extraction by skipping lowercase-copy work when content has no ASCII uppercase bytes; adds matching tests.
tgrep-core/src/query.rs	Reworks OR execution to union sorted results incrementally and adds posting-list sort/dedup helpers plus tests.
tgrep-core/src/ondisk.rs	Splits file-entry validation/writing so files.bin entries can be streamed without temporary allocation; adds round-trip test coverage.
tgrep-core/src/builder.rs	Replaces full-build inverted-map construction with batched flat postings, then writes index files from sorted postings.
tgrep-core/benches/trigram_extraction.rs	Adds Criterion benchmarks for trigram extraction on lowercase and mixed-case corpora.
tgrep-core/benches/query_execution.rs	Adds Criterion benchmarks for AND/OR execution and on-disk literal query workloads.
tgrep-core/benches/index_build.rs	Adds Criterion benchmarks for index-build throughput and a Windows-only peak working set probe.
tgrep-core/Cargo.toml	Adds benchmark dependencies and registers the new benchmark targets.
Cargo.toml	Bumps the workspace version.
Cargo.lock	Records new benchmark-related dependencies and updated crate versions.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 9/10 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

tgrep-core/benches/index_build.rs:150

• format_mib is only called from the Windows-specific peak-memory path, so it becomes dead code on non-Windows builds and causes the bench target to fail clippy -D warnings. This function should be gated the same way as its only caller.

fn format_mib(bytes: u64) -> f64 {
    bytes as f64 / 1024.0 / 1024.0
}

• Files reviewed: 11/12 changed files
• Comments generated: 5

[Copilot]

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 11/12 changed files
• Comments generated: 4

[Copilot]

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 12/13 changed files
• Comments generated: 3