Add the capability to speed up S3 uploads using AWS transfer manager

[suraj-goel]

Description

• Use AWS Transfer Manager to speed up uploads to S3.

Motivation

• To improve speed of uploads to S3.

[suraj-goel]

I will work on this issue.

[LakshSingla]

I was working on something similar a while back. Maybe these changes will help you #14834. What I found was that the speed improvements weren't a lot for MSQ because transfer manager in the SDK v1(?) didn't parallelize if the downloads were chunked. It helped a bit when segments were getting downloaded and uploaded. However, for some reason, segments uploaded without the transfer manager were slowly downloaded when downloaded with the transfer manager.
As such, it would be really cool if you can benchmark the performance of the changes, along with speedups/slowdown when downloaded and uploaded pre-post your patch.

[suraj-goel]

I was working on something similar a while back. Maybe these changes will help you #14834. What I found was that the speed improvements weren't a lot for MSQ because transfer manager in the SDK v1(?) didn't parallelize if the downloads were chunked. It helped a bit when segments were getting downloaded and uploaded. However, for some reason, segments uploaded without the transfer manager were slowly downloaded when downloaded with the transfer manager. As such, it would be really cool if you can benchmark the performance of the changes, along with speedups/slowdown when downloaded and uploaded pre-post your patch.

@LakshSingla, Thanks for the suggestions.
Right now, My changes are focussed on optimizing the uploads.
I will try to come up with some existing benchmarks or create a new one for transfer manager.