Skip to content

Add bit-packing #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
a10y opened this issue Oct 11, 2024 · 3 comments
Open

Add bit-packing #4

a10y opened this issue Oct 11, 2024 · 3 comments

Comments

@a10y
Copy link
Contributor

a10y commented Oct 11, 2024

Currently I just split, dict-encode etc. Need to wire up bit-packing (and unpacking)

@mwlon
Copy link

mwlon commented Mar 20, 2025

Do I understand right that this still isn't exposed in the API? I'm hoping for some sort of function (or chain of functions) that take me from e.g. &[f32] -> Vec<u8>.

@robert3005
Copy link
Member

For ALP-RD we need to bitpack te values which is not wired here yet. This is very specific to how ALP-RD is laid out internally and return type would be more complex than just Vec<u8>. Generally if your data is ALP encodeable you can run ALP on it and then use https://github.com/spiraldb/fastlanes for bitpacking. The full cascading compression is implemented in https://github.com/spiraldb/vortex/

@mwlon
Copy link

mwlon commented Mar 20, 2025

The split contains 5 things. Are you saying you apply fastlanes to each of them? I think a complete example is warranted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants