Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support to enable Map encoded custom forward index #15006

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

raghavyadav01
Copy link
Contributor

PR add support for enabling Custom MAP encoded forwarded index creator and reader.
Summary :

  • immutable Segment Index Build Path

Collect MapColumnPreIndexStatsCollector during gather stats phase
Collect MAP encoded column custom metadata properties file and merge with segment metadata.properties file.

  • Immutable Segment Loader Path

Initialize MAP column with child columns (keys)
Collect all the key level indexes into custom MAP column IndexContainer.
For MAP encoded columns initialize Custom Forward Index Reader with child column indexes buffer.

  • Couple of MAP data column bug fixes

Fix MAP convert for segment upload use case.
Fix Dictionary Index reader API .

@codecov-commenter
Copy link

codecov-commenter commented Feb 6, 2025

Codecov Report

Attention: Patch coverage is 21.36752% with 92 lines in your changes missing coverage. Please review.

Project coverage is 63.69%. Comparing base (59551e4) to head (3439a27).
Report is 1679 commits behind head on master.

Files with missing lines Patch % Lines
.../segment/index/column/MapColumnIndexContainer.java 0.00% 18 Missing ⚠️
...egment/spi/index/metadata/SegmentMetadataImpl.java 0.00% 12 Missing ⚠️
...converter/stats/RealtimeSegmentStatsContainer.java 25.00% 8 Missing and 1 partial ⚠️
...ment/index/forward/ForwardIndexCreatorFactory.java 11.11% 7 Missing and 1 partial ⚠️
...ment/creator/impl/SegmentColumnarIndexCreator.java 22.22% 5 Missing and 2 partials ⚠️
...segment/spi/index/metadata/ColumnMetadataImpl.java 0.00% 7 Missing ⚠️
...ndex/converter/SegmentV1V2ToV3FormatConverter.java 0.00% 5 Missing and 1 partial ⚠️
...he/pinot/segment/spi/index/ForwardIndexConfig.java 16.66% 5 Missing ⚠️
...indexsegment/immutable/ImmutableSegmentLoader.java 55.55% 3 Missing and 1 partial ⚠️
...a/org/apache/pinot/common/utils/PinotDataType.java 0.00% 3 Missing ⚠️
... and 7 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #15006      +/-   ##
============================================
+ Coverage     61.75%   63.69%   +1.94%     
- Complexity      207     1483    +1276     
============================================
  Files          2436     2717     +281     
  Lines        133233   152358   +19125     
  Branches      20636    23557    +2921     
============================================
+ Hits          82274    97046   +14772     
- Misses        44911    48001    +3090     
- Partials       6048     7311    +1263     
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) ⬆️
integration 100.00% <ø> (+99.99%) ⬆️
integration1 100.00% <ø> (+99.99%) ⬆️
integration2 0.00% <ø> (ø)
java-11 63.68% <21.36%> (+1.97%) ⬆️
java-21 63.54% <21.36%> (+1.91%) ⬆️
skip-bytebuffers-false 63.69% <21.36%> (+1.94%) ⬆️
skip-bytebuffers-true 63.52% <21.36%> (+35.80%) ⬆️
temurin 63.69% <21.36%> (+1.94%) ⬆️
unittests 63.69% <21.36%> (+1.94%) ⬆️
unittests1 56.19% <17.94%> (+9.30%) ⬆️
unittests2 34.02% <10.25%> (+6.29%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants