Commit 62b3ad3
authored
fix: return to hardcoded model IDs for Vertex AI (#4041)
# What does this PR do?
partial revert of b67aef2
Vertex AI doesn't offer an endpoint for listing models from Google's
Model Garden
Return to hardcoded values until such an endpoint is available
Closes #3988
## Test Plan
Server side, set up your Vertex AI env vars (`VERTEX_AI_PROJECT`,
`VERTEX_AI_LOCATION`, and `GOOGLE_APPLICATION_CREDENTIALS`) and run the
starter distribution
```bash
$ llama stack list-deps starter | xargs -L1 uv pip install
$ llama stack run starter
```
Client side, formerly broken cURL requests now working
```bash
$ curl http://127.0.0.1:8321/v1/models | jq '.data | map(select(.provider_id == "vertexai"))'
[
{
"identifier": "vertexai/vertex_ai/gemini-2.0-flash",
"provider_resource_id": "vertex_ai/gemini-2.0-flash",
"provider_id": "vertexai",
"type": "model",
"metadata": {},
"model_type": "llm"
},
{
"identifier": "vertexai/vertex_ai/gemini-2.5-flash",
"provider_resource_id": "vertex_ai/gemini-2.5-flash",
"provider_id": "vertexai",
"type": "model",
"metadata": {},
"model_type": "llm"
},
{
"identifier": "vertexai/vertex_ai/gemini-2.5-pro",
"provider_resource_id": "vertex_ai/gemini-2.5-pro",
"provider_id": "vertexai",
"type": "model",
"metadata": {},
"model_type": "llm"
}
]
$ curl -fsS http://127.0.0.1:8321/v1/openai/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"vertexai/vertex_a
i/gemini-2.5-flash\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}], \"max_tokens\": 128, \"temperature\": 0.0}" | jq
{
"id": "p8oIaYiQF8_PptQPo-GH8QQ",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"logprobs": null,
"message": {
"content": "Hello there! How can I help you today?",
"refusal": null,
"role": "assistant",
"annotations": null,
"audio": null,
"function_call": null,
"tool_calls": null
}
}
],
...
```
Signed-off-by: Nathan Weinberg <[email protected]>1 parent cb40da2 commit 62b3ad3
File tree
1 file changed
+10
-0
lines changed- src/llama_stack/providers/remote/inference/vertexai
1 file changed
+10
-0
lines changedLines changed: 10 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
0 commit comments