5854037 bytes

[XinyuZeng]

TL;DR Tried various methods but natebrennand's general approach still works well. The major improvements come from using LZMA to compress the repo column that yields better results than ZSTD.

• Add columnar storage for repository data with FoR encoding
• Implement varint delta encoding for sorted u64 values
• Add 24-bit packing/unpacking for u32 values
• Add patched bitpacking with outlier handling, but discarded
• Use LZMA for repo-related data compression
• Implement streaming decoding for varint delta and FoR values
• Sort events by id and use zstd level 22
• Various optimizations and cleanup

[agavra]

Hi @XinyuZeng - very neat! For the spirit of the competition I prefer that we avoid adding new dependencies outside of zstd (that way the focus is on the compression of the data itself instead of trying different bytewise compression libraries to shave a few bytes). I'm pretty certain that even with zstd insteadof LZMA you will be near the top of the leaderboard. Do you want to try that out?

I will update the competition instructions to verify this, I hope you learned a lot during the challenge though!

[XinyuZeng]

Yes, switching block compression isn't fancy at all! Just found the repo names is the major bottleneck to optimize so trying better string compression here. Closing this and going to vibe a new one if I have time!