ENH lazy load modules in the root __init__

[adrinjalali]

#824 requires modifying __getattr__ and __dir__ on the module level (available from py3.7: https://peps.python.org/pep-0562/)

This PR makes loading modules on the root __init__ of the repo to be lazy loaded, which also gives us a nice intro to implementing our own __getattr__ and __dir__ for futher changes.

As a side effect, this makes import huggingface_hub much faster:

on main:

$ time python -c "import huggingface_hub"
python -c "import huggingface_hub"  1.98s user 0.19s system 98% cpu 2.206 total

this PR:

$ time python -c "import huggingface_hub"
python -c "import huggingface_hub"  0.04s user 0.02s system 98% cpu 0.063 total

For more info: https://scientific-python.org/specs/spec-0001/

cc @LysandreJik @osanseviero @nateraw

also closes #875

[adrinjalali]

failing CI due to this. Corner case @tupui , @stefanv ? (Trying lazy loading in on this project, but this makes our doc builder to fail since it resolves this to the module rather than the function).

[stefanv]

I can reproduce this behavior, although I will need to dig a bit into the cause.

snapshot_download is a function after import, but then gets overwritten by the module of the same name upon first access.

[stefanv]

OK, so the problem here is that the module and the function have the same name, so when you try and access huggingface_hub.snapshot_download it is not clear what it should refer to. I can add a warning to our code for this situation, but you probably want to rename the module to _snapshot_download.py.

[stefanv]

I'd probably add something like this: https://github.com/adrinjalali/huggingface_hub/pull/1

[adrinjalali]

Yep, we agree with renaming the file (ref: #875), but if lazy loading is to be adopted by more major libraries, it probably should try to not change the behavior I think.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[adrinjalali]

CI is green now.

[LysandreJik]

This is very nice, thanks for working on it @adrinjalali!

Very clean to rely on the Python 3.7 feature, it's simpler to complete than what we have in transformers as we only need to add it to the resulting dependency directory.

[osanseviero]

This is very cool 🔥 thanks for the PR!

[adrinjalali]

I updated as you suggested @LysandreJik , but in general I would be more in favor of not changing them since it makes it harder for us to copy new versions from upstream.

This should be ready for merge then :) I'm excited for the next steps.

[LysandreJik]

I understand that changing them leads to more friction when wanting to update. Do you expect this code to evolve over time? I would prefer that all code in the codebase is formatted the same and documented the same, but I understand the pushback.

[stefanv]

The code is changing in response to this issue, in fact :) It highlighted a small bug when functions are implemented inside of modules with the same name.

But, in general, we don't expect many big changes.

[adrinjalali]

@LysandreJik the point of marking verdored code to me is to also make it clear that it's not our code, therefore it doesn't follow our standards or style, and it should not be exposed publicly.

Saying this, I now realize we're exposing this on the top level as a public function, I'll submit a PR making it private.