Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Host Snappy compression #17824

Open
wants to merge 12 commits into
base: branch-25.04
Choose a base branch
from
Open

Host Snappy compression #17824

wants to merge 12 commits into from

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Jan 25, 2025 •
edited
Loading

Description

Issue #17641

Simple implementation of the algorithm based on https://github.com/google/snappy/blob/main/snappy.cc.

Other changes:

  • Limited the number of threads in the compression thread pool to 32, based on benchmark results;
  • Submit tasks to the thread pool starting from the largest buffers to improve CPU utilization (impact verified with benchmark results);
  • Changed the environment variable from LIBCUDF_USE_HOST_COMPRESSION to LIBCUDF_HOST_COMPRESSION, and its type to string, to allow potential future settings "OFF", "AUTO" and "ON".

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jan 25, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Jan 25, 2025
@vuule vuule added feature request New feature or request non-breaking Non-breaking change labels Jan 25, 2025
@vuule vuule changed the base branch from branch-25.02 to branch-25.04 January 30, 2025 20:42
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Feb 4, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added the CMake CMake build issue label Feb 6, 2025
@vuule
Copy link
Contributor Author

vuule commented Feb 6, 2025

/ok to test

@vuule vuule marked this pull request as ready for review February 7, 2025 01:20
@vuule vuule requested review from a team as code owners February 7, 2025 01:20
@vuule vuule requested review from vyasr and lamarrr February 7, 2025 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vyasr vyasr Awaiting requested review from vyasr vyasr is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@lamarrr lamarrr Awaiting requested review from lamarrr lamarrr is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

At least 1 approving review is required to merge this pull request.

Assignees

@vuule vuule

Labels
CMake CMake build issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vuule