Skip to content

Conversation

@bentsherman
Copy link
Member

@bentsherman bentsherman commented Oct 21, 2025

This PR simplifies the new S3 config options that were added as part of the upgrade to AWS SDK v2, in preparation for the 25.10 release

Spun off from #6369

  • Replace transferManagerThreads config option with semaphores

    • transfer manager uses semaphore to limit copies and uploads to prevent out-of-memory error
    • transfer manager already uses semaphore to limit downloads based on download buffer size
  • Make the docs more user friendly

    • use aws targetThroughputInGbps to limit S3 transfers
    • use aws maxConnections to limit all other S3 API calls
    • remove maxConcurrency and maxNativeMemory from docs -- they will remain as hidden options for now

@bentsherman bentsherman added this to the 25.10 milestone Oct 21, 2025
@bentsherman bentsherman requested a review from jorgee October 21, 2025 15:30
@bentsherman bentsherman requested a review from a team as a code owner October 21, 2025 15:30
@netlify
Copy link

netlify bot commented Oct 21, 2025

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 00a12da
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/68f7d62ee1661a000832e898
😎 Deploy Preview https://deploy-preview-6496--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@bentsherman bentsherman force-pushed the simplify-s3-config-options branch from 8e29322 to 00a12da Compare October 21, 2025 18:51
@bentsherman bentsherman merged commit 0b7f180 into master Oct 21, 2025
23 checks passed
@bentsherman bentsherman deleted the simplify-s3-config-options branch October 21, 2025 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants