Implemented No-overwrite for uploads using cp command #9569
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kdaily
reviewed
Jun 24, 2025
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
a66905b to
69ea8ad
Compare
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
41b7423 to
8ad2ec2
Compare
kdaily
reviewed
Jun 26, 2025
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
e8f7134 to
7985693
Compare
kdaily
requested changes
Jul 8, 2025
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
35e8e12 to
68fb87d
Compare
aemous
requested changes
Jul 9, 2025
| @@ -195,6 +195,7 @@ class TransferManager: | |||
| + [ | |||
| 'ChecksumType', | |||
| 'MpuObjectSize', | |||
| 'IfNoneMatch', | |||
There was a problem hiding this comment.
Since we're modifying the vendored s3transfer, we should likely be porting the same changes to boto/s3transfer.
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
2 times, most recently
from
427869f to
274113a
Compare
kdaily
requested changes
Jul 9, 2025
| @@ -859,6 +859,49 @@ def test_can_specify_amount_for_nonseekable_stream(self): | |||
| self.assertEqual(nonseekable_fileobj.read(3), 'foo') | |||
|
|
|||
|
|
|||
| class TestRequestParamsMapperNoOverwrite(unittest.TestCase): | |||
| def test_set_no_overwrite_param_when_flag_present(self): | |||
| # Test when no_overwrite flag is present | |||
There was a problem hiding this comment.
nit: Since the test name describes it well, I don't think the inline comments are needed. I would only add them if there's something not captured in the test name or needs to be called out. Same with below tests.
There was a problem hiding this comment.
In last two test methods, I had to explicitly mention setting the parameter hence comments are still present and for rest it is removed
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
2 times, most recently
from
bf4d8a1 to
0dcab51
Compare
aemous
requested changes
Jul 10, 2025
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
c096558 to
e1c4d4a
Compare
aemous
requested changes
Jul 10, 2025
kdaily
requested changes
Jul 11, 2025
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
3 times, most recently
from
50a1015 to
c6b558f
Compare
added 2 commits
July 11, 2025 15:31
Updated s3Transfer manager generating original version Updated args list updated transfer manager for upload
Removed unwanted comments for test
rakshil14-2
force-pushed
the
no_overwrite_for_cp_command
branch
from
c6b558f to
6345224
Compare
kdaily
approved these changes
Jul 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue : Github Issue #2874
Description of changes:
Customers requested to prevent overwriting of objects. This is done by providing
no-overwriteheader usingifNoneMatchfeature of Amazon S3 which only allows to upload objects that are not present on the bucket. Theno-overwriteheader provides support to multi-part upload as well as works well with othercpcommand headers. Customer can be implemented usingaws s3 cp <source> <destination> --no-overwritewhich allows successful uploads when object with the same name is not present on the destination. However, if the user tries to upload the object which is already present at the target bucket using command line, awarningabout skipping the file will be generated and that object is not allowed to upload to S3 bucket.Testing:
Functional testing is performed to validate the object upload process. This included both scenarios uploading an object when object with same key is already present in the bucket as well as those when the object with same key is not present on bucket. These tests were conducted for both single-part uploads as well as multipart uploads.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.