Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add segment StarTree index rebuild throttler configurable via ZK cluster configs #14943

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

somandal
Copy link
Contributor

@somandal somandal commented Jan 30, 2025

Rebuilding the StarTree indexes can be more resource intensive than rebuilding other indexes. This PR adds a specific StarTree index rebuild throttler as a safety net to limit the parallel StarTree index rebuilds that can occur.

This change is related to #14894 and utilizes the same ZK cluster config listener mechanism for config updates.

This PR adds the following:

  • Adds a CLUSTER configuration to control the max startree preprocess parallelism allowed
  • Adds a SegmentStarTreePreprocessThrottler class to control how many startree index rebuild operations can happen in parallel. This is plugged into the ImmutableSegmentLoader and SegmentPreProcesser on code paths where startree index rebuild throttling is desirable

I had considered adding the above to the existing SegmentPreprocessThrottler class itself, but wanted to avoid confusion regarding which semaphore should be used where in the code. Using a different class allows for better isolation of concerns.

cc @Jackie-Jiang @klsince @npawar

@somandal somandal changed the title Add ZK cluster configurable segment StarTree index rebuild throttler Add segment StarTree index rebuild throttler configurable via ZK cluster configs Jan 30, 2025
@codecov-commenter
Copy link

codecov-commenter commented Jan 30, 2025

Codecov Report

Attention: Patch coverage is 74.11765% with 22 lines in your changes missing coverage. Please review.

Project coverage is 63.70%. Comparing base (59551e4) to head (e6225e5).
Report is 1648 commits behind head on master.

Files with missing lines Patch % Lines
...ocal/utils/SegmentStarTreePreprocessThrottler.java 77.08% 9 Missing and 2 partials ⚠️
.../pinot/server/starter/helix/BaseServerStarter.java 0.00% 5 Missing ⚠️
...ata/manager/realtime/RealtimeTableDataManager.java 0.00% 2 Missing ⚠️
...ocal/segment/index/loader/SegmentPreProcessor.java 87.50% 0 Missing and 2 partials ⚠️
...rg/apache/pinot/server/starter/ServerInstance.java 0.00% 1 Missing ⚠️
...server/starter/helix/HelixInstanceDataManager.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14943      +/-   ##
============================================
+ Coverage     61.75%   63.70%   +1.95%     
- Complexity      207     1472    +1265     
============================================
  Files          2436     2711     +275     
  Lines        133233   151989   +18756     
  Branches      20636    23470    +2834     
============================================
+ Hits          82274    96832   +14558     
- Misses        44911    47884    +2973     
- Partials       6048     7273    +1225     
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) ⬆️
integration 100.00% <ø> (+99.99%) ⬆️
integration1 100.00% <ø> (+99.99%) ⬆️
integration2 0.00% <ø> (ø)
java-11 63.68% <74.11%> (+1.97%) ⬆️
java-21 63.58% <74.11%> (+1.96%) ⬆️
skip-bytebuffers-false 63.70% <74.11%> (+1.95%) ⬆️
skip-bytebuffers-true 63.56% <74.11%> (+35.83%) ⬆️
temurin 63.70% <74.11%> (+1.95%) ⬆️
unittests 63.70% <74.11%> (+1.95%) ⬆️
unittests1 56.22% <30.76%> (+9.32%) ⬆️
unittests2 34.02% <68.23%> (+6.29%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants