Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cleanup): Implement pauseless table segment cleanup logic #15002

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

9aman
Copy link
Contributor

@9aman 9aman commented Feb 6, 2025

Behavior Details

The endpoint resets segment state for pauseless-enabled tables in two modes:

Mode 1: Explicit Error Segments

When error segments are provided via the segments query parameter:

  • Deletes only the specified error segments
  • Returns detailed segment names in response if ≤ 5 segments, otherwise returns count

Mode 2: Automatic Detection

When no error segments are provided, for each partition:

  • Automatically detects error segments from ExternalView
  • Identifies the oldest error segment (lowest sequence number) in that partition
  • Deletes all segments in that partition with sequence numbers >= the oldest error segment

Retention Period Handling

The retention period for deleted segments follows this precedence:

  1. Input parameter (via retention query param)
  2. Table config
  3. Cluster setting
  4. 7d (default)

Note: Use 0d or -1d for immediate deletion without retention

API: DELETE /resetPauselessTable/{tableName}
Flags: ?type=[OFFLINE|REALTIME]&retention=[period]&segments=[list]

| Seq | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|-----|---|---|---|---|---|---|---|---|---|
| Seg | █ | █ | █ | █ | █ | █ | █ | █ | █ |
| Act | ✓ | ✓ | ✓ | ✂️ | ✓ | ✓ | ✓ |✂️ | ✓ |

User-provided error segments: {3, 7}


When no segments are provided in input: 
Sequence:    0    1    2    3    4    5    6    7    8
             ▼    ▼    ▼    ▼    ▼    ▼    ▼    ▼    ▼
Segments:    █    █    █    █    █    █    █    █    █
             |SAFE ZONE (seq 0–4)  |  DELETION ZONE (seq 5–8)|
Error:                                🔴 (Error at seq=5: oldest error)

9aman added 2 commits February 6, 2025 11:18
This commit implements a cleanup mechanism for pauseless tables that:
1. Takes error segments as input and finds the oldest error segment
   (lowest sequence number) per partition
2. For each partition, deletes all segments with sequence numbers
   greater than or equal to its oldest error segment.

API: DELETE /resetPauselessTable/{tableName}
Flags: ?type=[OFFLINE|REALTIME]&retention=[period]&segments=[list]
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 30.64516% with 43 lines in your changes missing coverage. Please review.

Project coverage is 63.67%. Comparing base (59551e4) to head (1145f3d).
Report is 1678 commits behind head on master.

Files with missing lines Patch % Lines
...ler/api/resources/PinotSegmentRestletResource.java 30.64% 41 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #15002      +/-   ##
============================================
+ Coverage     61.75%   63.67%   +1.92%     
- Complexity      207     1483    +1276     
============================================
  Files          2436     2716     +280     
  Lines        133233   152295   +19062     
  Branches      20636    23544    +2908     
============================================
+ Hits          82274    96980   +14706     
- Misses        44911    48021    +3110     
- Partials       6048     7294    +1246     
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) ⬆️
integration 100.00% <ø> (+99.99%) ⬆️
integration1 100.00% <ø> (+99.99%) ⬆️
integration2 0.00% <ø> (ø)
java-11 63.63% <30.64%> (+1.93%) ⬆️
java-21 63.56% <30.64%> (+1.94%) ⬆️
skip-bytebuffers-false 63.65% <30.64%> (+1.90%) ⬆️
skip-bytebuffers-true 63.54% <30.64%> (+35.81%) ⬆️
temurin 63.67% <30.64%> (+1.92%) ⬆️
unittests 63.67% <30.64%> (+1.92%) ⬆️
unittests1 56.21% <ø> (+9.32%) ⬆️
unittests2 34.01% <30.64%> (+6.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants