[ENH] Make roll dirty log always converge to coalesce everything.

[rescrv]

Description of changes

The dirty log was not guaranteed to converge even in the absence of
further writes. I confirmed this empirically with local testing and
inspection of the dirty log. Post-patch it always converges within one
rollup.

Test plan

CI

• Tests pass locally with pytest for python, yarn test for js, cargo nextest for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Make Dirty Log Rollup Deterministically Converge and Refactor Log Compaction Path

This PR overhauls the dirty log rollup and compaction mechanics in the Rust log service, making dirty log coalescence deterministic: after a single rollup, the dirty log always converges regardless of further write activity. The changes extend to multi-shard rollup concurrency, cache management for coalesced cursor witnesses, test improvements, and several minor refactors and parameter tunings.

Key Changes:
• Redesigns dirty log rollup and compaction to ensure convergence after a single rollup, regardless of further incoming markers
• Refactors DirtyMarker::coalesce_markers API to operate on external mutable rollups and forget sets, improving testability and clarity of state mutation
• Ensures consistent detection and removal of purged collections from rollups across concurrent streams during rollup
• Extends Rollup, RollupPerCollection, and transient rollup processing to handle concurrent dirty marker reduction (buffered parallelism)
• Improves cache integration for cursor witness objects, correctly serializing/deserializing witness state to accelerate rollup and compaction checks
• Fixes and parameterizes related tests and utilities (Python property-based add test for minimal add, log offsets test for nonintermittent CI behavior)
• Bumps relevant operational tuning parameters (e.g., increases compactor concurrency and I/O queue limits to reduce compaction latency)
• Updates the CursorStore and Witness types in wal3 for easier, explicit witness/cursor access
• Miscellaneous documentation and error handling improvements

Affected Areas:
• rust/log-service/src/lib.rs (core rollup/convergence logic)
• rust/wal3/src/cursors.rs (witness/cursor struct refactor)
• rust/wal3/src/writer.rs (concurrency/gc for log writer)
• chromadb/test/property/test_add.py (property-based test tuning)
• rust/log/tests/log-offsets.rs (integration test deflake)
• rust/worker/tilt_config.yaml (operational param tuning)
• rust/log-service/src/bin/chroma-inspect-dirty-log.rs (debug tool buffer size)
• rust/log/src/grpc_log.rs (minor error string fix)

This summary was automatically generated by @propel-code-bot

[Sicheng-Pan]

magic constant. Why do we take the first k (and in what order)?

[rescrv]

I left this in by mistake.

[Sicheng-Pan]

q: why not coalesce_markers(&records, &mut rollup.rollups, &mut rollup.forget) or maybe coalesce_marksers(&records, &mut rollup)?

[rescrv]

Forget needs to be maintained across futures. Will document.