Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Update ScyllaDB required dependencies for compression #169

Merged
merged 2 commits into from
Feb 11, 2025

Conversation

EXPEbdodla
Copy link
Collaborator

@EXPEbdodla EXPEbdodla commented Feb 11, 2025

What this PR does / why we need it:

Which issue(s) this PR fixes:

  1. Scylladb Python Driver supports compression by default. Need necessary packages installed to support that. Doc. Code Docs
  2. Added Scylladb dependencies to CI requirements
  3. Updated other dependencies (mypy and singlestoredb restrictions are from feast)
  4. Rest all are formatting issues due to ruff dependency upgrade

Misc

@EXPEbdodla EXPEbdodla changed the title fix: Update ScyllaDB required dependencies fix: Update ScyllaDB required dependencies for compression Feb 11, 2025
@@ -115,12 +115,14 @@

CASSANDRA_REQUIRED = [
"cassandra-driver>=3.24.0,<4",
"lz4",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will save storage space and costs related to that, but can it potentially increase retrieval latencies?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it also increase CPU load?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All our tests used anaconda image and these dependencies are installed automatically. With switching from anaconda, these needs to be explicitly added.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh so we were already using compression then?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. With latest Dataproc images, we loose those dependencies.

@EXPEbdodla EXPEbdodla merged commit 6b2643a into master Feb 11, 2025
25 checks passed
@EXPEbdodla EXPEbdodla deleted the update_dependencies branch February 11, 2025 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants