Add segment StarTree index rebuild throttler configurable via ZK cluster configs #14943
+632
−113
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebuilding the StarTree indexes can be more resource intensive than rebuilding other indexes. This PR adds a specific StarTree index rebuild throttler as a safety net to limit the parallel StarTree index rebuilds that can occur.
This change is related to #14894 and utilizes the same ZK cluster config listener mechanism for config updates.
This PR adds the following:
SegmentStarTreePreprocessThrottler
class to control how many startree index rebuild operations can happen in parallel. This is plugged into theImmutableSegmentLoader
andSegmentPreProcesser
on code paths where startree index rebuild throttling is desirableI had considered adding the above to the existing
SegmentPreprocessThrottler
class itself, but wanted to avoid confusion regarding which semaphore should be used where in the code. Using a different class allows for better isolation of concerns.cc @Jackie-Jiang @klsince @npawar