-
Notifications
You must be signed in to change notification settings - Fork 653
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support List[str]
in InferenceClient.feature_extraction
text
parameter
#2824
Comments
Hi @anakin87 , thanks for opening this issue and for describing your findings. I agree with you that supporting For the record and as mentioned in the PR above, I have pushed back in the past for this feature request (see #1745 and #1746 (comment)) but I think it's time to revisit. I'll let you know once it's done but you can't already consider it safe and future-proof to use it like this. EDIT: and glad to see the deprecation warning is been carefully listened too ❤️ |
Related to huggingface/huggingface_hub#2824. This PR makes it possible to send a `string[]` instead of `string` as `feature-extraction` inputs. This is already possible in practice in Inference API but not documented. In the past, I've pushed back on this change (see huggingface/huggingface_hub#1745 and huggingface/huggingface_hub#1746 (comment)) but I think it's fine to revisit it now. The main reason I mentioned was that `feature-extraction`'s server-side implementation was mostly a for-loop on the text input so acception a `string[]` would not really improve performances. That been said, there has been quite some improvements since then and especially the `text-embedding-inference` framework.
Hello and thanks for your work!
Context
I work on the Haystack LLM framework, which provides Embedders based on
huggingface_hub
.The
HuggingFaceAPITextEmbedder
transforms a string into a vector. TheHuggingFaceAPIDocumentEmbedder
enriches a Document with its embedding.These components allow using different HF APIs via
huggingface_hub
: Serverless Inference API, Inference Endpoints, and self-hosted Text Embeddings Inference container.In the past, we used
InferenceClient.post
to query the API, mainly to supporttruncate
andnormalize
(which were missing in thefeature_extraction
method before #2270).Now that
InferenceClient.post
is changing and will be removed in 0.31.0, I'm migrating toInferenceClient.feature_extraction
.Issue
There's only one issue:
In
HuggingFaceAPIDocumentEmbedder
, we allow batching, which is explicitly supported by TEI (and Inference Endpoints).Passing a list of inputs to the
text
parameter offeature_extraction
seems to work correctly—I've tested it with Serverless Inference API, Inference Endpoints, and self-hosted TEI.However, the method signature of
feature_extraction
officially supports onlystr
, notList[str]
.Questions
Given this, I'd like to ask:
List[str]
infeature_extraction
?InferenceClient
?The text was updated successfully, but these errors were encountered: