Skip to content

Updating multi LLM samples without precaching #657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Updating multi LLM samples without precaching #657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions config/samples/nim/serving/standalone/no-precaching/multi-llm-hf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
# NIM Service with Multi-LLM NIM with Autoscaling
# NIM Service with Multi-LLM NIM with model not pre-cached
apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
Expand All @@ -12,10 +12,12 @@ spec:
pullPolicy: IfNotPresent
pullSecrets:
- ngc-secret
authSecret: hf-secret # with HF_TOKEN set
authSecret: ngc-api-secret # with NGC_API_KEY set
env:
- name: NIM_MODEL_NAME
value: hf://meta-llama/Llama-3.2-1B-Instruct
- name: HF_TOKEN
Copy link
Collaborator

@shivamerla shivamerla Sep 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to be part of secret as well, in this case we should not make NGC_API_KEY checks in the code mandatory and export all env from the secret instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes thats the fix. But this PR is just to fix the sample for v3.0.0 release.

value: <your-hf-token> # Replace with your actual HF token
storage:
pvc:
create: true
Expand Down
6 changes: 3 additions & 3 deletions config/samples/nim/serving/standalone/no-precaching/multi-llm-ngc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
# NIM Service with Multi-LLM NIM with Autoscaling
# NIM Service with Multi-LLM NIM with model not pre-cached
apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
name: meta-llama-3-2-1b-instruct
name: meta-llama-3-8b-instruct
namespace: nim-service
spec:
image:
Expand All @@ -15,7 +15,7 @@ spec:
authSecret: ngc-api-secret
env:
- name: NIM_MODEL_NAME
value: nvidia/nemo/llama-3_2-1b-instruct
value: 'ngc://nvidian/nim-llm-dev/meta-llama3-8b-instruct:hf'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is internal NGC repository. Let's remove it for now? thanks

storage:
pvc:
create: true
Expand Down
Loading