Overview

This repostiory is desigined to add KubeAI to any ConfigSync enabled Kubernetes cluster. The manifests generated provides default confiuration for GKE and Google Distributed Cloud clusters with GPUs enabled. Select the approprate folder under /config or /package. The two folders contain the same contents, the only differnt file names and if they are applied using Fleet Packages vs manual or via Primary Root Repository.

How to use

There are two methods to applying this package to your cluster(s). The first is using ConfigSync as outlined below. The only decision needed is the type of cluster you are deploying to. The package provides 2 variants: GKE and GDC with GDC being a Google Distributed Cloud cluster with GPUs installed and enabled.

If you fork this repository and run under a private repository, the example describes how an ExternalSecret can be created allowing the RootSync to authenticate to a private repository.

⚠️ This can be applied into any ConfigSync cluster using the RootSync below, otherwise apply manually with kubectl apply -f /config/<cluster-type>/*.yaml. This can be embedded into the Primary Root Repository for multi-cluster fleet deployment.

With OUT Authentication

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: kubeai-cluster-trait-sync
  namespace: config-management-system
  annotations:
    configsync.gke.io/deletion-propagation-policy: Foreground
spec:
  sourceFormat: "unstructured"
  git:
    repo: "https://gitlab.com/gcp-solutions-public/retail-edge/available-cluster-traits/kubeai-cluster-trait"
    branch: "main"
    dir: "/config/<GKE OR GDC>/"
    auth: "none"

With Git Authentication (preferred)

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: kubeai-cluster-trait-sync
  namespace: config-management-system
  annotations:
    configsync.gke.io/deletion-propagation-policy: Foreground
spec:
  sourceFormat: "unstructured"
  git:
    repo: "https://github.com/gitlab.com/gcp-solutions-public/retail-edge/available-cluster-traits/kubeai-cluster-trait"
    branch: "main"
    dir: "/config<GKE OR GDC>/"
    auth: "token"
    secretRef:
      name: kube-ai-git-creds               # matches the ExternalSecret spec.target.name below

---

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kube-ai-git-creds-es
  namespace: config-management-system
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-secret-store
  target:                                       # K8s secret definition
    name: kube-ai-git-creds                     ############# Matches the secretRef above
    creationPolicy: Owner
  data:
  - secretKey: username                         # K8s secret key name inside secret
    remoteRef:
      key: kube-ai-cluster-trait-git-creds      #  GCP Secret Name
      property: username                        # field inside GCP Secret
  - secretKey: token                            # K8s secret key name inside secret
    remoteRef:
      key: kube-ai-cluster-trait-git-creds      #  GCP Secret Name
      property: token                           # field inside GCP Secret

Deploy without ConfigSync

# if using GKE
kubectl apply -f package/kube-ai-cluster-trait-pkg-gke.yaml

# if using GDC
kubectl apply -f package/kube-ai-cluster-trait-pkg-gdc.yaml

Create GCP Secret for git-creds

Create the GCP Secret Manager secret used by ExternalSecret to proxy for K8s Secret

export PROJECT_ID=<your google project id>
export SCM_TOKEN_TOKEN=<your gitlab personal-access token value>
export SCM TOKEN_USER=<your gitlab personal-access token user>

gcloud secrets create kube-ai-cluster-trait-git-creds --replication-policy="automatic" --project="${PROJECT_ID}"
echo -n "{\"token\"{{':'}} \"${SCM_TOKEN_TOKEN}\", \"username\"{{':'}} \"${SCM_TOKEN_USER}\"}" | gcloud secrets versions add kube-ai-cluster-trait-git-creds --project="${PROJECT_ID}" --data-file=-

Find new version of helm chart

Using Helm, add the repo, update and then describe the chart to get the latest version.

helm repo add kubeai https://www.kubeai.org
helm repo update
helm show chart kubeai/kubeai

Local Validation

Assuming nomos is installed (via gcloud components install nomos)

nomos vet --no-api-server-check --path config/

Docker method

Using this link to find the version of nomos-docker: https://cloud.google.com/anthos-config-management/docs/how-to/updating-private-registry#expandable-1

docker pull gcr.io/config-management-release/nomos:stable
docker run -it -v $(pwd):/code/ gcr.io/config-management-release/nomos:stable nomos vet --no-api-server-check --path /code/config/

ConfigSync Overview

See our documentation for how to use each subdirectory.