Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve S3 upload speeds using aws transfer manager #17674

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

suraj-goel
Copy link
Contributor

@suraj-goel suraj-goel commented Jan 28, 2025

Fixes #17669.

Description

  • Improve S3 Upload speeds using AWS Transfer Manager.
  • Benchmark Results:

Screenshot 2025-01-28 at 11 45 04 AM

  • As per the benchmark results, minimumUploadPartSize/multipartUploadThreshold should be set to 20 MB for best performance.

Release note

Improved S3 Upload Performance in Druid

  • Integrated S3 Transfer Manager to enhance the speed and efficiency of S3 uploads within Druid. This update significantly reduces upload times of segments to S3.
  • The feature can be enabled by adding following configs in common.runtime.properties
    druid.storage.transfer.useTransferManager=true
    druid.storage.transfer.minimumUploadPartSize=20971520
    druid.storage.transfer.multipartUploadThreshold=20971520

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@@ -201,7 +201,7 @@

private MockAmazonS3Client()
{
super(new AmazonS3Client(), new NoopServerSideEncryption());
super(new AmazonS3Client(), new NoopServerSideEncryption(), new S3TransferConfig());

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation Note test

Invoking
AmazonS3Client.AmazonS3Client
should be avoided because it has been deprecated.
@suraj-goel suraj-goel marked this pull request as ready for review January 28, 2025 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add the capability to speed up S3 uploads using AWS transfer manager
1 participant