Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions #678

omarespejel · 2022-02-10T01:27:51Z

Sorry, I created a new PR (previous one: PR huggingface/huggingface_hub#416) I had a couple of mistakes with git, and seemed easier to create a new one. Thanks @muellerzr and @osanseviero for your comments!

What

Add fastai upstream and downstream capabilities for versions fastai>=2.4 (link) and fastcore>=1.3.27 (link).
Inform users that lower versions of fastai are not supported yet (TODO -- Examine whether it is worth implementing previous versions).

Why

Supporting version fastai1 might not be worth it. Loading and pushing Learners involves several changes and the fastai1 library has not been updated for over a year.
Supporting versions 2.0.6>= fastai <2.4 involve complex changes. Fastai was updated due to what appear to be modifications to pytorch/serialization.py. For the sake of agility in our first release, I suggest releasing this version without supporting these previous versions.
fastai>=2.4 versions work with fastcore>=1.3.27 version which is installed automatically when fastai>=2.4 is installed.

How?

Following huggingface_hub practices, file_download.py checks for fastcore and fastai availability and versions.
The fastai and fastcore versions used to train the Learner are automatically stored in a config.json when pushed to hub.
A README.md is automatically generated, if none, with the fastai tag.

Testing?

Tested that all fastai>=2.4 versions work with the present code and with the fastcore==1.3.27 version.

Next?

Examine whether it is worth implementing fastai <2.4 versions. I will ask on the fastai forums to assess how many users would require this support.
Update description on how to load fastai models in Libraries.ts.

…brary

muellerzr

Very nice! I just left a few nits :)

src/huggingface_hub/fastai_mixin.py

osanseviero

Thank you very much for the PR! There are some important things to address, but overall is going in right direction 🚀

src/huggingface_hub/fastai_mixin.py

…e name from fastai_mixin

…gingface_hub into add-fastai-mixin

omarespejel · 2022-02-14T03:38:55Z

Hi, @osanseviero @muellerzr thank you for your kind reviews guys! All feedback was applied and tested.

Main changes:

Changed name. From fastai_mixin.py to fastai_utils.py.
No longer a mixin. Eliminated FastaiModelHubMixin class and only leaved three main public functions: from_pretrained_fastai, push_to_hub_fastai, and save_fastai_learner. This helps with simplicity.
Imported fastai_utils.py public functions in huggingface_hub/__init__.py.
Failing test. Build (3.6) and Build (3.9) actions failing only due to the test_inference_with_audio test. Will review the reason.
Created check_fastai_fastcore_versions function to not force fastai imports from the beggining.

Will be here :D

osanseviero · 2022-02-15T09:27:06Z

The failing tests are unrelated. They are flaky and likely not related to this PR, so don't worry about it

osanseviero

Looking good! I think it's in a good state to start adding some tests 😄

src/huggingface_hub/fastai_utils.py

…licit

…tion

* fastai was made compatible with Python 3.10 just in the most recent version 3.5.6 released on April 1st. I would prefer to wait for the next release of fastai.

… test_fastai_integration.py

* Additionally, isort test_fastai_integration.py * Change the name from save_fastai_learner to _save_pretrained_fastai * isort __init__.py

* functions from_pretrained_fastai and push_to_hub_fastai were added to mixins.mdx

src/huggingface_hub/fastai_utils.py

osanseviero

Cool 🚀 Thanks for iterating on this

src/huggingface_hub/fastai_utils.py

…rectory * By default learner.export saves learner to learner.path * Before we where saving the model in learner.path and them moving it to save_directory * Fix by changing learner.path to being equal to save_directory

* Additionally, makes optional that the pyproject.toml contains the fastai and fastcore versions * Will continue to throw an error if the toml library is not available * Will continue to thrown an error if the pyproject.toml specifies fastai or fastcore versions that are not supported by from_pretrained_fastai * This will allow to load fastai models from the Hub that were not necessarily uploaded using push_to_hub_fastai.

…and "requires" section

…stcore versions

…s after warnings

* This allows to eliminate the returns when checking for the versions of fastai and fastcore in the pyproject.toml

…gingface_hub into add-fastai-mixin

…into add-fastai-mixin

osanseviero · 2022-04-26T18:33:52Z

Congrats @omarespejel! 🚀 🚀 🚀

omarespejel · 2022-04-27T19:07:38Z

Thank you @osanseviero @adrinjalali @muellerzr @nateraw! For your reviews and feedback 🤗🥳

omarespejel added 4 commits October 17, 2021 14:24

added a fastai mixin for upstream and downstream tasks with fastai li…

f237986

…brary

Added check for right fastai version

16a10a2

Added aditional documentation

a0398d9

Add support for fastai>=2.4 versions and fastcore>=1.3.27

0e65896

omarespejel added the enhancement New feature or request label Feb 10, 2022

omarespejel requested review from osanseviero and muellerzr February 10, 2022 01:27

omarespejel self-assigned this Feb 10, 2022

omarespejel changed the title ~~Add fastai mixin~~ Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions Feb 10, 2022

Merge branch 'main' into add-fastai-mixin

aea8b8e

muellerzr previously requested changes Feb 10, 2022

View reviewed changes

src/huggingface_hub/fastai_mixin.py Outdated Show resolved Hide resolved

src/huggingface_hub/fastai_mixin.py Outdated Show resolved Hide resolved

src/huggingface_hub/fastai_mixin.py Outdated Show resolved Hide resolved

src/huggingface_hub/fastai_mixin.py Outdated Show resolved Hide resolved

osanseviero reviewed Feb 10, 2022

View reviewed changes

osanseviero requested a review from nateraw February 10, 2022 17:14

omarespejel added 7 commits February 11, 2022 09:49

Update docstrings, save_fastai_learner, push_to_hub_fastai, and chang…

940c68d

…e name from fastai_mixin

Merge branch 'add-fastai-mixin' of https://github.com/omarespejel/hug…

4b246f4

…gingface_hub into add-fastai-mixin

trim triling whitespaces

807e7bb

Import fastai_utils.py in huggingface_hub/__init__py

44493cf

Add check_fastai_fastcore_versions function in fastai_utils.py

a7253ed

Eliminate imports of libraries not used in fastai_utils.py

5904986

Add isort and black format to fastai_utils.py

18b4498

omarespejel requested review from osanseviero and muellerzr February 14, 2022 03:39

osanseviero reviewed Feb 15, 2022

View reviewed changes

omarespejel added 5 commits February 15, 2022 13:52

Change pickle_protocol argument from kwargs to explicit

37b6b51

Change kwargs arguments in the function from_pretrained_fastai to exp…

1c9cff5

…licit

Simplify push_to_hub_fastai function, particularly the repo_id argument

2a40a8d

Eliminate search for a pickle document in from_pretrained_fastai func…

1a4012c

…tion

Simplify push_to_hub_fastai and correct bug in from_pretrained_fastai

a9ba5d2

omarespejel added 6 commits April 12, 2022 18:56

black format fastai_utils.py

022c571

Come back to Python 3.9 instead of 3.10

5dafb25

* fastai was made compatible with Python 3.10 just in the most recent version 3.5.6 released on April 1st. I would prefer to wait for the next release of fastai.

Change name name of save_fastai_learner to _save_pretrained_fastai in…

9a06a67

… test_fastai_integration.py

Change the name to _save_pretrained_fastai in __init__.py

004d318

* Additionally, isort test_fastai_integration.py * Change the name from save_fastai_learner to _save_pretrained_fastai * isort __init__.py

Fix nits in test_fastai_integration.py

947d58b

Add fastai integration to docs

d6ee8bf

* functions from_pretrained_fastai and push_to_hub_fastai were added to mixins.mdx

osanseviero reviewed Apr 20, 2022

View reviewed changes

src/huggingface_hub/fastai_utils.py Outdated Show resolved Hide resolved

Fix nits

18d0d27

omarespejel mentioned this pull request Apr 21, 2022

Changes suggested to fastai integration #848

Open

3 tasks

osanseviero approved these changes Apr 21, 2022

View reviewed changes

omarespejel and others added 14 commits April 21, 2022 11:42

Fix wording

7e63f46

black fastai_utils.py

8633c0e

Add warnings if the pyproject.toml does not contain a "build-system" …

ac0ef8a

…and "requires" section

Change try-excepts for ifs in the warnings checking the fastai and fa…

2b28f4b

…stcore versions

Fix nits in documentation

88f4cd0

Move errors in _check_fastai_fastcore_pyproject_versions to condition…

d0268a9

…s after warnings

Misc improvements

5ffd543

Move the versions checks for fastai and fastcore to "else"'s

f36093e

* This allows to eliminate the returns when checking for the versions of fastai and fastcore in the pyproject.toml

Move the versions checks for fastai and fastcore to "else"'s

cd4648a

* This allows to eliminate the returns when checking for the versions of fastai and fastcore in the pyproject.toml

Merge branch 'add-fastai-mixin' of https://github.com/omarespejel/hug…

1795ce8

…gingface_hub into add-fastai-mixin

Merge branch 'main' of https://github.com/huggingface/huggingface_hub …

f9dd4b0

…into add-fastai-mixin

Black reformat

465e0fa

omarespejel requested a review from osanseviero April 26, 2022 18:18

osanseviero approved these changes Apr 26, 2022

View reviewed changes

osanseviero merged commit cab4152 into huggingface:main Apr 26, 2022

omarespejel deleted the add-fastai-mixin branch April 26, 2022 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions #678

Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions #678

omarespejel commented Feb 10, 2022 •

edited

Loading

muellerzr left a comment

osanseviero left a comment

omarespejel commented Feb 14, 2022

osanseviero commented Feb 15, 2022

osanseviero left a comment

osanseviero left a comment

osanseviero commented Apr 26, 2022

omarespejel commented Apr 27, 2022

Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions #678

Add fastai upstream and downstream capacities for fastai>=2.4 and fastcore>=1.3.27 versions #678

Conversation

omarespejel commented Feb 10, 2022 • edited Loading

What

Why

How?

Testing?

Next?

muellerzr left a comment

Choose a reason for hiding this comment

osanseviero left a comment

Choose a reason for hiding this comment

omarespejel commented Feb 14, 2022

osanseviero commented Feb 15, 2022

osanseviero left a comment

Choose a reason for hiding this comment

osanseviero left a comment

Choose a reason for hiding this comment

osanseviero commented Apr 26, 2022

omarespejel commented Apr 27, 2022

omarespejel commented Feb 10, 2022 •

edited

Loading