[1.0 launch] BBR & BBR helm image from RC work

[kfswain]

Part of #1454

[kfswain]

@capri-xiyue or @zetxqx pls consider picking this up!

[capri-xiyue]

Syned offline. I don't think currently @zetxqx and I have the bandwidth. Other contributors feel free to pick this up.

[chewong]

I can help validate!
/assign

[chewong]

1. Install Gateway API and Gateway API Inference Extension CRDs

• Gateway API: v1.3.0
• Gateway API Inference Extension v1.0.0-rc.1

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml --server-side=true
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.0-rc.1/manifests.yaml

2. Install Istio Gateway with inference.networking.k8s.io/v1 InferencePool support

Based on this commit from @danehans:

TAG=1.28-alpha.45e4f30947d52a3710f2b11e11c208dee247fa3a
HUB=gcr.io/istio-testing
helm upgrade -i istio-base oci://$HUB/charts/base --version $TAG -n istio-system --create-namespace
helm upgrade -i istiod oci://$HUB/charts/istiod \
  --version $TAG \
  -n istio-system \
  --set meshConfig.defaultConfig.proxyMetadata.ENABLE_GATEWAY_API_INFERENCE_EXTENSION="true" \
  --set pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION="true" \
  --set tag=$TAG \
  --set hub=$HUB \
  --wait

3. Deploy Istio Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP

4. Deploy EPP

I have deployed two vLLM model servers at port 5000 ahead of time:

1. microsoft/Phi-4-mini-instruct with label app=phi-4-mini-instruct
2. mistralai/Mistral-7B-Instruct-v0.3 with label app=mistral-7b-instruct

Deploy v1 InferencePool for each model server.

helm upgrade --install phi-inferencepool \
  --set inferencePool.modelServers.matchLabels.app=phi-4-mini-instruct \
  --set "inferencePool.targetPorts[0].number"=5000 \
  --version v1.0.0-rc.1 \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool

helm upgrade --install mistral-inferencepool \
  --set inferencePool.modelServers.matchLabels.app=mistral-7b-instruct \
  --set "inferencePool.targetPorts[0].number"=5000 \
  --version v1.0.0-rc.1 \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool

Deploy Istio DestinationRules for each model server due to self-signed certificates of EPP services.

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: phi-inferencepool-epp-insecure-tls
spec:
  host: phi-inferencepool-epp
  trafficPolicy:
    tls:
      mode: SIMPLE
      insecureSkipVerify: true
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: mistral-inferencepool-epp-insecure-tls
spec:
  host: mistral-inferencepool-epp
  trafficPolicy:
    tls:
      mode: SIMPLE
      insecureSkipVerify: true

5. Deploy BBR

helm upgrade --install body-based-router \
  --set provider.name=istio \
  --version v1.0.0-rc.1 \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/body-based-routing

6. Deploy HTTPRoute

HTTPRoute with inference.networking.k8s.io/v1 InferencePool backendRefs.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - matches:
    - headers:
      - type: Exact
        name: X-Gateway-Model-Name
        value: phi-4-mini-instruct
      path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: phi-inferencepool
      group: inference.networking.k8s.io
      kind: InferencePool
  - matches:
    - headers:
      - type: Exact
        name: X-Gateway-Model-Name
        value: mistral-7b-instruct
      path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: mistral-inferencepool
      group: inference.networking.k8s.io
      kind: InferencePool

6a. Test Chat Completions

Phi-4-mini-instruct

export CLUSTERIP=$(kubectl get svc inference-gateway-istio -o jsonpath='{.spec.clusterIP}')
kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "phi-4-mini-instruct",
    "messages": [{"role": "user", "content": "What is kubernetes?"}],
    "max_tokens": 50
  }' | jq

Response

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "message": {
        "annotations": null,
        "audio": null,
        "content": "Kubernetes, often abbreviated as K8s, is an open-source platform designed to automate the deployment, scaling, and operations of containerized applications. Developed by Google and donated to the Cloud Native Computing Foundation (CNCF), Kubernetes provides an orches",
        "function_call": null,
        "reasoning_content": null,
        "refusal": null,
        "role": "assistant",
        "tool_calls": []
      },
      "stop_reason": null
    }
  ],
  "created": 1756320162,
  "id": "chatcmpl-a6f2a84f-a46e-4da0-a95c-98c0ec1723c9",
  "kv_transfer_params": null,
  "model": "phi-4-mini-instruct",
  "object": "chat.completion",
  "prompt_logprobs": null,
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 17,
    "prompt_tokens_details": null,
    "total_tokens": 67
  }
}

---

Mistral-7b-instruct

export CLUSTERIP=$(kubectl get svc inference-gateway-istio -o jsonpath='{.spec.clusterIP}')
kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-7b-instruct",
    "messages": [{"role": "user", "content": "What is kubernetes?"}],
    "max_tokens": 50
  }' | jq

Response

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "message": {
        "annotations": null,
        "audio": null,
        "content": " Kubernetes, often known as K8s, is an open-source platform designed to automate deploying, scaling, and managing containerized applications. It groups containers that make up an application into logical units for easy management and orchestration.",
        "function_call": null,
        "reasoning_content": null,
        "refusal": null,
        "role": "assistant",
        "tool_calls": []
      },
      "stop_reason": null
    }
  ],
  "created": 1756320060,
  "id": "chatcmpl-e8a0874d-77ce-4c95-b160-cd501cd7d75f",
  "kv_transfer_params": null,
  "model": "mistral-7b-instruct",
  "object": "chat.completion",
  "prompt_logprobs": null,
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 8,
    "prompt_tokens_details": null,
    "total_tokens": 58
  }
}

6b. Test Completions

Phi-4-mini-instruct

export CLUSTERIP=$(kubectl get svc inference-gateway-istio -o jsonpath='{.spec.clusterIP}')
kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "phi-4-mini-instruct",
    "prompt": "What is kubernetes?",
    "max_tokens": 50
  }' | jq

Response

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "prompt_logprobs": null,
      "stop_reason": null,
      "text": " Kubernetes is an open-source platform designed to automate deploying, scaling, and managing containerized applications. Developed by Google and now maintained by the Cloud Native Computing Foundation, it is a container orchestration system that helps in automating tasks and easing the management of"
    }
  ],
  "created": 1756320434,
  "id": "cmpl-fd24c0d9-3dcb-4e95-ac91-190f14a043a5",
  "kv_transfer_params": null,
  "model": "phi-4-mini-instruct",
  "object": "text_completion",
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 6,
    "prompt_tokens_details": null,
    "total_tokens": 56
  }
}

---

Mistral-7b-instruct

export CLUSTERIP=$(kubectl get svc inference-gateway-istio -o jsonpath='{.spec.clusterIP}')
kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST http://$CLUSTERIP/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-7b-instruct",
    "prompt": "What is kubernetes?",
    "max_tokens": 50
  }' | jq

Response

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "prompt_logprobs": null,
      "stop_reason": null,
      "text": "\n\nKubernetes is an open-source container-orchestration system for automating the deployment, scaling, and management of containerized applications. It was originally developed by Google, and is now maintained by the Cloud Native Computing Foundation (CN"
    }
  ],
  "created": 1756320494,
  "id": "cmpl-6e664edd-03db-4fe3-ab95-a42588f253cd",
  "kv_transfer_params": null,
  "model": "mistral-7b-instruct",
  "object": "text_completion",
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 6,
    "prompt_tokens_details": null,
    "total_tokens": 56
  }
}

Suggestions

Automate Istio DestinationRule creation for InferencePool resources. When using Istio as the provider (provider.name == istio), the InferencePool controller should automatically create corresponding DestinationRules to handle TLS configuration for EPP services. This would eliminate the manual step of creating DestinationRules for self-signed certificates. (#1488)

[kfswain]

Thanks @chewong! If this isnt in our guides on the site: https://gateway-api-inference-extension.sigs.k8s.io/guides/ do you mind adding this there so we can consider this resolved? Thanks!

The guides directory for the site is here: https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/site-src/guides