Skip to content

[EPIC] Port object_store content from arrow-rs repository #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
6 tasks done
alamb opened this issue Mar 20, 2025 · 21 comments
Closed
6 tasks done

[EPIC] Port object_store content from arrow-rs repository #2

alamb opened this issue Mar 20, 2025 · 21 comments
Assignees
Labels
development-process Related to development process of object-store

Comments

@alamb
Copy link
Contributor

alamb commented Mar 20, 2025

This epic tracks the items needed to move the object_store code from https://github.com/apache/arrow-rs to this repo

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

I will begin moving the code now

alamb pushed a commit to alamb/arrow-rs-object-store that referenced this issue Mar 20, 2025
* Implement bulk_delete_request for Azure

* Fix lint and add Azurite bug workaround

* Special 404 error case

* Clippy fix

* Make number of expected headers more conservative and better document invariants

* Use multer for multipart parsing

* Fix clippy

* Fix clippy apache#2

* Reuse part response buffer

* Make multer conditional to azure feature

* One more HeaderValue::from_static

* Add tests for bulk delete request building and response parsing

* Switch back to manual parsing to avoid multer dependency, other PR suggestions

* Fixes lint
@alamb alamb self-assigned this Mar 20, 2025
@assignUser
Copy link
Member

@alamb do you want to move only open issues or closed issues as well? Caveat, if a label on an issue doesn't exists in the target repo it is dropped. I have done this for go and java with the gh cli

This was referenced Mar 20, 2025
@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

@alamb do you want to move only open issues or closed issues as well? Caveat, if a label on an issue doesn't exists in the target repo it is dropped. I have done this for go and java with the gh cli

Moving closed issues would be amazing.

@assignUser do you have the scripts you used (or can you point me at them) for how you migrated the issues using gh?

@assignUser
Copy link
Member

assignUser commented Mar 20, 2025

gh issue list -R apache/arrow-rs --label "object-store" -L 1000 --state "all" --json 'number' -q '.[].number' |  xargs -I% gh issue transfer % https://github.com/apache/arrow-rs-object-store

This will transfer all 248 issues with the "object-store" label to this repo. Any references to these issues in PRs etc. will break afaik. As we have those in commit messages in the monorepo we only transfered open issues but from what I can see this is not the case for you. They will also be re-numbered

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

Thanks @assignUser

I ran the following commands:

gh issue list -R apache/arrow-rs --label "object-store" -L 1000 --state "all" --json 'number' -q '.[].number' > object_store_prs.txt

And spot checked a few random numbers in to make sure they were actually issues with object store (which they were)

Which look good

I am now goign to play around with a few tests issues and see how the process works

$ cat test_issues.txt
2561
2550
2373
2361

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

To test it I did the following to move one ticket:

Leave a comment

$ cat test_issues.txt | head -n 1 | xargs -I% gh -R apache/arrow-rs issue comment % -b "Migrated from https://github.com/apache/arrow-rs/issues/%"
https://github.com/apache/arrow-rs/issues/2561#issuecomment-2741640115

Then I ran this to move the actual ticket

$ cat test_issues.txt | head -n 1 | xargs -I% gh -R apache/arrow-rs issue transfer % https://github.com/apache/arrow-rs-object-store
https://github.com/apache/arrow-rs-object-store/issues/9

And the new location looks good:

I also verified that the old URL still works:

I will now try with a few more and if they look good I'll migrate them all

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

cat object_store_issues.txt | xargs -I% gh -R apache/arrow-rs issue comment % -b "Migrating from arrow-rs issue [#%](https://github.com/apache/arrow-rs/issues/%)"

Resulted in these comments:

Details

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

Thanks for your help @assignUser -- the commands seem to have worked great!

@assignUser
Copy link
Member

Happy to help :)

I also verified that the old URL still works:

Oh cool I didn't know that!

@ByteBaker
Copy link
Contributor

I think we should also remove "object store" from the issues.

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

I think we should also remove "object store" from the issues.

Thanks @ByteBaker -- do you mean remove the object_store label?

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

@alamb
Copy link
Contributor Author

alamb commented Mar 20, 2025

And a few more:

#293
#294
#295
#296
#297
#298
#299
#300
#301

@ByteBaker
Copy link
Contributor

ByteBaker commented Mar 20, 2025

Thanks @ByteBaker -- do you mean remove the object_store label?

@alamb No. I didn't see any labels to begin with. I was referring to the issue titles. E.g.:

[object_store] Runtime Agnostic Retry Machinery

You'd notice a lot of issues have this or simply the two words Object Store in the title. It made sense earlier since it was all inside arrow-rs, but now they're all related to object store only. So no point having the title contain that.

@assignUser
Copy link
Member

Here is a script that'll re-title those issues:

gh issue list -R apache/arrow-rs-object-store -L 400 --json 'title,number' -q '[.[] | select(.title | test("^\\[?object[-_ ]?store\\]?:?.*"; "i"))] | .[].title |= sub("^\\[?object[-_ ]?store\\]?:? *"; ""; "i")| .[]| [.number, .title] | @tsv' |
while IFS=$'\t' read -r number title; do
  gh issue edit -R apache/arrow-rs-object-store "$number" -t "$title"
done

@alamb
Copy link
Contributor Author

alamb commented Mar 21, 2025

You'd notice a lot of issues have this or simply the two words Object Store in the title. It made sense earlier since it was all inside arrow-rs, but now they're all related to object store only. So no point having the title contain that.

Cool -- yeah if someone wants to run that it is cool. I don't mind the redundancy -- maybe someone can find time to go through the tickets manually sometime and close ones / clean them up if they don't make sense

@ByteBaker
Copy link
Contributor

@alamb I'd be happy to, but I don't have the right access.

@alamb
Copy link
Contributor Author

alamb commented Mar 21, 2025

Thanks @ByteBaker

@ByteBaker
Copy link
Contributor

Thanks @ByteBaker

Ohh, I meant I can't modify since I don't have permissions. 😅 @alamb

@alamb
Copy link
Contributor Author

alamb commented Mar 24, 2025

I think this issue is now done 🎉 Thanks everyone

@alamb alamb closed this as completed Mar 24, 2025
@alamb alamb added the development-process Related to development process of object-store label May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development-process Related to development process of object-store
Projects
None yet
Development

No branches or pull requests

3 participants