[ENH] Batch get info from sysdb for compaction scheduler

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Refactors verify_and_enrich function in compaction scheduler so that it will get information from sysdb in batch (vs one by one previously)
  • The compaction scheduler no longer panics on collections with broken offsets. Instead, an error trace will be emitted and the collection will be ignored for compaction
• New functionality
  • N/A

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[Sicheng-Pan]

• [ENH] Batch get info from sysdb for compaction scheduler #4926 👈 (View in Graphite)
• [ENH] Purge dirty log in background at the end of scheduled compaction #4915
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Refactor Compaction Scheduler to Batch Fetch from sysdb & Improve Offset Handling

This PR refactors the compaction scheduler's collection verification and enrichment to perform batch retrievals from sysdb instead of one-by-one queries, significantly improving efficiency. It also replaces scheduler panics on broken log offsets with error logging and skips such collections from compaction, enhancing service robustness and reliability.

Key Changes:
• verify_and_enrich_collections now fetches collection and tenant info in batches from sysdb using HashMaps for mapping.
• Broken offset scenarios no longer trigger panics; impacted collections are ignored with error-level tracing.
• Unit test updated: replaced #[should_panic]-based panic check with a test that asserts no jobs are scheduled for broken offsets.
• Minor improvements to tracing and error handling throughout the scheduler.

Affected Areas:
• rust/worker/src/compactor/scheduler.rs (core logic and related tests)

This summary was automatically generated by @propel-code-bot

[Sicheng-Pan]

Note: the payload of the batch response might be too large. consider setting a limit for the batch size

[propel-code-bot]

[BestPractice]

When we know the collection is empty and need to avoid panicking in the first_log_offset case, consider setting a limit for the batch size as suggested in the PR comments. Adding a batch size limit would prevent potential payload size issues when fetching collections with many records.

Suggested change

	collection_ids: Some(collection_id_to_log_info.keys().cloned().collect()),
	..Default::default()
	})
	collection_ids: Some(collection_id_to_log_info.keys().cloned().collect()),
	limit: Some(1000), // Add reasonable batch limit to prevent oversized payloads
	..Default::default()

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.