Add support for ListObjectsV2

[Ankk98]

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

[geemus]

Similar here, we likely either overlooked it or it was added later. I'd certainly welcome a PR to add it if you have the time and inclination, and I'm certainly happy to support that however I can.

[Ankk98]

Here is a quick idea on how ListObjectsV2 support can be added while maintaining V1 compatibility:

I will add:
A new request file: lib/fog/aws/requests/storage/list_objects_v2.rb
A new parser file: lib/fog/aws/parsers/storage/list_objects_v2.rb

Then we will be able to use it like:

storage.list_objects_v2('bucket-name', {
  'prefix' => 'my/prefix/',
  'max-keys' => 1000,
  'fetch-owner' => true
})

Optionally, we can add an option, use_v2 = true in:
Directories#get
Files#all
Directory#objects_keys
and other places where this is being used.

Is this fine?

[geemus]

@Ankk98 hmm, something along those lines sounds reasonable. Could you summarize the differences between v1 and v2, if you know them? I looked at the doc you linked, but it wasn't immediately obvious to me. Depending on how it has changed, it might make sense to make v2 the default for models, but I was having a hard time figuring out what was different.

[Ankk98]

Key Differences

1. Pagination

• ListObjects

  • Uses marker parameter for pagination
  • Returns NextMarker in response
  • More complex pagination logic

• ListObjectsV2

  • Uses continuation-token parameter for pagination
  • Returns NextContinuationToken in response
  • Simpler and more efficient pagination mechanism

2. Response Structure

• ListObjectsV2
  • Has additional fields like StartAfter and KeyCount

3. Performance

ListObjectsV2

• Better performance
• Optimized for large buckets
• More efficient pagination

AWS recommends to use v2 API as default.

[geemus]

Thanks for the updates, I think on reflection that your approach sounds like a good one.

I was debating whether we should instead have v2 be the new default for models and allow opting back into v1, but I worry that could inadvertently lead to breakage for people. So this is probably safer for now.

[Ankk98]

Yeah, I think a 2 step approach will be good.
In the first PR we can add support for v2.
And later we can make v2 as default after some time in a different release.

If this sounds good to you, I can send a pr.

[geemus]

@Ankk98 Sure, sounds good to me, thanks!