Update inference API specification to include new Llama Service #5020
Conversation
Jan-Kazlouski-elastic
requested review from
jonathan-buttner
and removed request for
a team
| * | ||
| * Create an inference endpoint to perform an inference task with the `llama` service. | ||
| * @rest_spec_name inference.put_llama | ||
| * @availability stack since=9.2.0 stability=stable visibility=public |
There was a problem hiding this comment.
@jonathan-buttner could you please check if this 9.2.0 version is correctly set here. I assume it is, but want to be sure.
There was a problem hiding this comment.
Yep this is correct 👍
…ng-completion # Conflicts: # output/openapi/elasticsearch-openapi.json # output/openapi/elasticsearch-serverless-openapi.json # output/schema/schema.json
|
Following you can find the validation changes against the target branch for the APIs. No changes detected. You can validate these APIs yourself by using the |
Jan-Kazlouski-elastic
added
the
skip-backport
|
@jonathan-buttner since we didn't have backport for Llama integration, I added skip-backport label in order to skip backport label github action check |
| * After creating the inference model, you cannot change the associated API key. | ||
| * If you want to use a different API key, delete the inference model and recreate it with the same name and the updated API key. | ||
| */ | ||
| api_key?: string |
There was a problem hiding this comment.
I should have mentioned this on the elasticsearch PR. When did you need to supply an API key? In my testing of running the stack locally I didn't need to supply one 🤔
I was running it like this:
PUT _inference/text_embedding/llama-text-embedding
{
"service": "llama",
"service_settings": {
"url": "http://localhost:8321/v1/inference/embeddings",
"model_id": "all-MiniLM-L6-v2"
}
}
There was a problem hiding this comment.
It is good point to discuss. Glad you're bringing that up.
Llama Stack doesn't have built in authorization check by default, so It is possible to use it without providing any tokens. Specially when testing with Ollama locally.
However for me it is doubtful that users are going to use llama stack without auth 100% of the time so I added this api_key parameter as option for clients that would want to set up bearer auth. Authentication Configuration is described in Distribution Overview's Server Configuration section of official Llama Stack guide.
https://llama-stack.readthedocs.io/en/latest/distributions/configuration.html#authentication-configuration
I haven't investigated it in depth but I think it is safe to assume that providing ability to send bearer token pretty much covers security concerns.
| * | ||
| * Create an inference endpoint to perform an inference task with the `llama` service. | ||
| * @rest_spec_name inference.put_llama | ||
| * @availability stack since=9.2.0 stability=stable visibility=public |
There was a problem hiding this comment.
Yep this is correct 👍
This PR adds changes to specification caused by elastic/elasticsearch#130092:
Additional actions