Document integration requirements

[adrinjalali]

Taken from: huggingface/hub-docs#62 (comment), leaving it here since the repos are now separate.

Something we were just talking with @LysandreJik is that at the moment we don't have documentation on specifications of each integration.

Trying to explain:

Many of our integrations have been happening with close collaboration with third party developers. We would sometimes change something in the backend to support certain aspects of the new integration. For instance, the fastai integration adds dependencies to pyproject.toml and then those dependencies are installed somewhere, somehow, before the model is loaded.

What's missing is the kind of documentation explaining what exactly is required from a fastai repo on our side, for it to be properly interpreted and the inference API to work on it. It would be nice for a PR such as #678 to add that kind of documentation before being merged. This would also allow others to implement their own integration, even in a different language than Python if they need to. It would also give clarity to users on how their repos should look like.

[osanseviero]

I like this idea!

At one point there was an idea of having a LIbraries page (internal issue: https://github.com/huggingface/moon-landing/issues/749) which would hold this information. Since most of the 3rd party library integrations are not within hub-docs, I don't think this should be documented here, but rather in huggingface_hub (except maybe for the 2 libraries integrated in huggingface_hub, and even then I would prefer all of them in a single place). WDYT?

[adrinjalali]

I agree that the documentation should be where the implementation is. But isn't that api-inference-community then?

[osanseviero]

As discussed offline, api-inference-community is focused in the API implementation and docker images, not in downstream and upstream integrations.

I think we will likely have a new libraries section in hf.co/docs/hub with a page for each library. I think that would be part of huggingface/hub-docs#62 (comment) and with the revamp of the Hub docs cc @NimaBoscarino