[TST] More benchmark queries for regex

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • This PR adds more regex patterns in the benchmark. The benchmark also serve as an integration for regex as it compares the result with bruteforce evaluation.
  • Updates a few dependencies. Verified that there should be no breaking change
  • Updates some wal3 test because fragment size changed after dependency. The existing fragment should be compatible and manifest should still be valid
• New functionality
  • N/A

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[Sicheng-Pan]

• [TST] More benchmark queries for regex #4910 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Benchmark Expansion for Regex Patterns and Rust Dataset; Arrow/Parquet Ecosystem Updates

This PR makes major changes to the benchmarking/integration suite for regex and fulltext search functionality by expanding the coverage and diversity of regular expressions used in tests and benchmarks, primarily focusing on realistic Rust code patterns. It introduces and wires up a new Rust dataset based on the BigCode 'the-stack-dedup' corpus, reworks the relevant dataset loader logic for async streaming from HuggingFace/hf-hub, and adapts the benchmark routines to leverage this dataset. There is also a substantial update to the Arrow and Parquet-related dependencies (from 52.x to 55.1) across the workspace, resulting in a large Cargo.lock diff and a need to update some file size assertions in integration tests. Additionally, dependency and dataset wiring is improved for future extensibility.

Key Changes:
• Adds comprehensive lists of realistic fulltext and regex query patterns to rust/index/benches/literal.rs and rust/worker/benches/regex.rs.
• Replaces ad-hoc filesystem dataset loading with a new loader for the BigCode The Stack deduplicated Rust subset (TheStackDedupRust), integrating async streaming of content.
• Adapts benchmark benchmarks and test frameworks to utilize the new Rust dataset for higher-fidelity evaluation.
• Upgrades Arrow, Parquet, and related dependencies to version 55.1 throughout the workspace and synchronizes code, test, and Cargo.lock for these changes.
• Corrects integration test fixtures involving file sizes/fragments for k8s/WAL tests to reflect new Parquet logic and block sizes.
• General dependency hygiene and cargo workspace corrections (e.g., moving to async Parquet APIs, resolving breaking changes, updating dataset utilities).

Affected Areas:
• Benchmark and test harnesses for regex/fulltext (rust/index/benches/literal.rs, rust/worker/benches/regex.rs)
• Dataset infrastructure (rust/benchmark/src/datasets/rust.rs, wiring in mod.rs and other dataset utilities)
• Dependency versions and build system (Cargo.toml, Cargo.lock, workspace settings)
• WAL3 integration/k8s tests expecting specific Parquet file sizes
• Dependency wiring for async Parquet/Arrow IO

This summary was automatically generated by @propel-code-bot