Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload large folders with symlinks #2883

Open
neverix opened this issue Feb 21, 2025 · 1 comment
Open

Upload large folders with symlinks #2883

neverix opened this issue Feb 21, 2025 · 1 comment

Comments

@neverix
Copy link

neverix commented Feb 21, 2025

Is your feature request related to a problem? Please describe.
I have a folder with symlinks. upload_large_folder currently doesn't work with that.

Describe the solution you'd like

(path.relative_to(folder_path).as_posix() for path in folder_path.glob("**/*") if path.is_file()),
Set recurse_symlinks to True here or add an option to enable that
Describe alternatives you've considered
Make it possible to upload_large_folder into paths other than the root. There's no other way to upload multiple large folders to one repo.

@hanouticelina
Copy link
Contributor

Hello @neverix, thanks for the feature request!

Set recurse_symlinks to True here or add an option to enable that

Adding symlink support seems feasible by adding a parameter to control Path.glob() behavior. We'll investigate if there are any limitations or additional maintenance burden, so let's keep this issue open 😄

Make it possible to upload_large_folder into paths other than the root. There's no other way to upload multiple large folders to one repo.

Regarding uploading to subfolders support: This limitation is intentional. The caching system that enables resuming interrupted uploads relies on a direct 1:1 relationship between local files and their destination in the repo. For now, we recommend structuring your local folders to match your desired repo structure.

In any case, upload_large_folder is primarily designed to be a reliable, resumable, and "resilient to errors" upload tool without having to compromise its robustness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants