[ENH]: More concurrent blockfilewriter

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Blockfilewriter was effectively single-threaded since it took a big fat mutex in the beginning of operations (set/delete/get_owned)
  • This PR makes it more performant through a combination of Optimistic Concurrency Control and Pessimistic Concurrency Control
  • The fat mutex is replaced by a partitioned mutex. The writer retries if any state changes concurrently. The entire set of Sparse index operations (splits and conversion to delta) are applied atomically now. The other details of the algorithm are deferred to reading of code
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

None

[sanketkedia]

• [ENH]: Add ability to set different block sizes for different blockfiles #4948
• [ENH]: granular lock for posting list #4937
• [ENH]: More concurrent blockfilewriter #4889 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Enable High-Concurrency for Arrow BlockfileWriter with Partitioned Mutexes

This PR overhauls concurrency control in the Arrow blockfile writer, replacing a single global mutex that serialized all write operations with a partitioned per-block locking scheme (using AsyncPartitionedMutex) and a hybrid of Optimistic and Pessimistic Concurrency Control. Major API changes are applied to set, delete, and get_owned operations to enable parallelism per block, with retries and atomic operations to avoid race conditions and ensure the correctness of block splits and sparse index updates. The SparseIndexWriter gains an 'apply_updates' method for atomic multi-block updates, and concurrency testing receives configuration improvements.

Key Changes:
• Replaces a single global (fat) write mutex in ArrowUnorderedBlockfileWriter with a partitioned async mutex keyed by block ID for finer-grained concurrent access.
• Rewrites set, delete, and get_owned methods to use lock-per-block and implement OCC/PCC: operation retries if concurrent index changes are detected.
• Introduces atomic multi-block sparse index updates via SparseIndexWriter::apply_updates, ensuring safe block splits and block replacements.
• Updates concurrency and unit tests to improve test coverage and confirm high-concurrency scenarios, including more aggressive block splitting.
• Refactors imports and test helpers to support the new locking primitives.

Affected Areas:
• rust/blockstore/src/arrow/blockfile.rs
• rust/blockstore/src/arrow/sparse_index.rs
• rust/blockstore/src/arrow/concurrency_test.rs

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[BestPractice]

There's a typo in the import name: AysncPartitionedMutex should be AsyncPartitionedMutex (missing the first 'n' in Async). This is present in several places in your code.

[propel-code-bot]

[BestPractice]

Great work implementing the partitioned mutex approach! This will significantly improve concurrency by allowing multiple operations on different blocks to proceed in parallel. One minor suggestion: consider adding a brief comment explaining the OCC/PCC approach near where the mutex is declared to help future developers understand the implementation.