Skip to content

fix(Helm)!: Refresh helm #880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Jul 1, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b22c8ff
Add tests to ensure service/ingress reliability
DiamondJoseph Jun 26, 2025
7f6db96
chore: Consolidate ConfigMaps
DiamondJoseph Jun 26, 2025
09bd101
Refresh service and ingress
DiamondJoseph Jun 26, 2025
fd19e3e
Add automount to service account
DiamondJoseph Jun 26, 2025
5c4c5f7
Extra documentation nice to haves
DiamondJoseph Jun 26, 2025
1ccb17f
Allow helm test to operate and be configured with externals
DiamondJoseph Jun 26, 2025
d8259bf
Add probes, allow arbitrary mounts, general improvements
DiamondJoseph Jun 26, 2025
b87ddf5
Fix tests
DiamondJoseph Jun 26, 2025
cecf616
Fix tests that relied on old Ingress name
DiamondJoseph Jun 26, 2025
8c2f287
Merge branch 'main' into refresh-helm
DiamondJoseph Jun 26, 2025
95cc1c2
Configure Helm docs as a chart
DiamondJoseph Jun 27, 2025
c384692
Merge branch 'main' into refresh-helm
DiamondJoseph Jun 27, 2025
20592e3
Adjust to change in default config
DiamondJoseph Jun 27, 2025
764b411
Merge branch 'main' into refresh-helm
DiamondJoseph Jun 30, 2025
d02760b
Merge commit '3f6fd8c5fd127e9f2daed0e51f2111afff7d2f34' into refresh-…
DiamondJoseph Jun 30, 2025
0553dce
Merge commit '4f2b41373d0c1a0fe690a8402f53922a0a27785d' into refresh-…
DiamondJoseph Jun 30, 2025
87faaf4
Add startupProbe ot allow devices to connect
DiamondJoseph Jun 30, 2025
daf2461
Merge commit '764b4119c820ca2bab67bc4da4a5192b8ab479c3' into refresh-…
DiamondJoseph Jun 30, 2025
a693fce
pebcak
DiamondJoseph Jun 30, 2025
94de2cb
More documentation and defaults from review
DiamondJoseph Jun 30, 2025
16cb19d
Re-enable readinessProbe
DiamondJoseph Jul 1, 2025
5bb0927
Re-enable readinessProbe
DiamondJoseph Jul 1, 2025
a275362
Merge commit '16cb19d48930572d00cc6f4d9d3bcbd75eb61e32' into refresh-…
DiamondJoseph Jul 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,11 @@ repos:
entry: ruff format --force-exclude
types: [python]
require_serial: true

- repo: https://github.com/norwoodj/helm-docs
rev: ""
hooks:
- id: helm-docs-container
args:
# Make the tool search for charts only under the `helm` directory
- --chart-search-root=helm
11 changes: 7 additions & 4 deletions helm/blueapi/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: v2
name: blueapi
home: https://github.com/DiamondLightSource/blueapi
description: A helm chart deploying a worker pod that runs Bluesky plans
description: A Helm chart deploying a worker pod that runs Bluesky plans

# A chart can be either an 'application' or a 'library' chart.
#
Expand All @@ -13,8 +13,11 @@ description: A helm chart deploying a worker pod that runs Bluesky plans
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# This is the chart version. This version number is incremented by the release process.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
# This is updated automatically by CI, the appVersion is automatically inserted by CI
version: 0.1.0

# This is the version number of the application being deployed. This version number is incremented by the release process.
# Versions are not expected to follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "0.1.0"
55 changes: 55 additions & 0 deletions helm/blueapi/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# blueapi

A Helm chart deploying a worker pod that runs Bluesky plans

**Homepage:** <https://github.com/DiamondLightSource/blueapi>

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | May be required to run on specific nodes (e.g. the control machine) |
| debug.enabled | bool | `false` | If enabled, disables liveness and readiness probes, and does not start the service on startup This allows connecting to the pod and starting the service manually to allow debugging on the cluster |
| extraEnvVars | list | `[]` | Additional envVars to mount to the pod |
| fullnameOverride | string | `""` | |
| hostNetwork | bool | `false` | May be needed for EPICS depending on gateway configuration |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"ghcr.io/diamondlightsource/blueapi"` | To use a container image that extends the blueapi one, set it here |
| image.tag | string | `""` | |
| imagePullSecrets | list | `[]` | |
| ingress | object | `{"annotations":{},"className":"nginx","enabled":false,"hosts":[{"host":"example.diamond.ac.uk","paths":[{"path":"/","pathType":"Prefix"}]}],"tls":[]}` | Configuring and enabling an ingress allows blueapi to be served at a nicer address, e.g. ixx-blueapi.diamond.ac.uk |
| ingress.hosts[0] | object | `{"host":"example.diamond.ac.uk","paths":[{"path":"/","pathType":"Prefix"}]}` | Request a host from https://jira.diamond.ac.uk/servicedesk/customer/portal/2/create/91 of the form ixx-blueapi.diamond.ac.uk. Note: pathType: Prefix is required in Diamond's clusters |
| initContainer | object | `{"enabled":false,"persistentVolume":{"enabled":false,"existingClaimName":""}}` | Configure the initContainer that checks out the scratch configuration repositories |
| initContainer.persistentVolume.enabled | bool | `false` | Whether to use a persistent volume in the cluster or check out onto the mounted host filesystem If persistentVolume.enabled: False, mounts scratch.root as scratch.root in the container |
| initContainer.persistentVolume.existingClaimName | string | `""` | May be set to an existing persistent volume claim to re-use the volume, else a new one is created for each blueapi release |
| livenessProbe | object | `{"failureThreshold":3,"httpGet":{"path":"/healthz","port":"http"},"periodSeconds":10}` | Liveness probe, if configured kubernetes will kill the pod and start a new one if failed consecutively. This is automatically disabled when in debug mode. |
| nameOverride | string | `""` | |
| nodeSelector | object | `{}` | May be required to run on specific nodes (e.g. the control machine) |
| podAnnotations | object | `{}` | |
| podLabels | object | `{}` | |
| podSecurityContext | object | `{}` | |
| readinessProbe | object | `{"failureThreshold":2,"httpGet":{"path":"/healthz","port":"http"},"periodSeconds":10}` | Readiness probe, if configured kubernetes will not route traffic to this pod if failed consecutively. This could allow the service time to recover if it is being overwhelmed by traffic, but without the to ability to load balance or scale up/outwards, upstream services will need to know to back off. This is automatically disabled when in debug mode. |
| resources | object | `{"limits":{"cpu":"2000m","memory":"4000Mi"},"requests":{"cpu":"200m","memory":"400Mi"}}` | Sets the compute resources available to the pod. These defaults are appropriate when using debug mode or an internal PVC and therefore running VS Code server in the pod. In the Diamond cluster, requests must be >= 0.1*limits When not using either of the above, the limits may be lowered. When idle but connected, blueapi consumes ~400MB of memory and 1% cpu and may struggle when allocated less. |
| restartOnConfigChange | bool | `true` | If enabled the blueapi pod will restart on changes to `worker` |
| securityContext.runAsNonRoot | bool | `true` | |
| securityContext.runAsUser | int | `1000` | |
| service.port | int | `80` | |
| service.type | string | `"ClusterIP"` | To make blueapi available on an IP outside of the cluster prior to an Ingress being created, change this to LoadBalancer |
| serviceAccount.annotations | object | `{}` | |
| serviceAccount.automount | bool | `true` | |
| serviceAccount.create | bool | `false` | |
| serviceAccount.name | string | `""` | |
| startupProbe | object | `{"failureThreshold":5,"httpGet":{"path":"/healthz","port":"http"},"periodSeconds":10}` | A more lenient livenessProbe to allow the service to start fully. This is automatically disabled when in debug mode. |
| tolerations | list | `[]` | May be required to run on specific nodes (e.g. the control machine) |
| tracing | object | `{"otlp":{"enabled":false,"protocol":"http/protobuf","server":{"host":"http://opentelemetry-collector.tracing","port":4318}}}` | Configure tracing: opentelemetry-collector.tracing should be available in all Diamond clusters |
| volumeMounts | list | `[{"mountPath":"/config","name":"worker-config","readOnly":true}]` | Additional volumeMounts on the output StatefulSet definition. Define how volumes are mounted to the container referenced by using the same name. |
| volumes | list | `[]` | Additional volumes on the output StatefulSet definition. Define volumes from e.g. Secrets, ConfigMaps or the Filesystem |
| worker | object | `{"api":{"url":"http://0.0.0.0:8000/"},"env":{"sources":[{"kind":"planFunctions","module":"dodal.plans"},{"kind":"planFunctions","module":"dodal.plan_stubs.wrapped"}]},"logging":{"graylog":{"enabled":false,"url":"http://graylog-log-target.diamond.ac.uk:12232/"},"level":"INFO"},"scratch":{"repositories":[],"root":"/blueapi-plugins/scratch"},"stomp":{"auth":{"password":"guest","username":"guest"},"enabled":false,"url":"http://rabbitmq:61613/"}}` | Config for the worker goes here, will be mounted into a config file |
| worker.api.url | string | `"http://0.0.0.0:8000/"` | 0.0.0.0 required to allow non-loopback traffic If using hostNetwork, the port must be free on the host |
| worker.env.sources | list | `[{"kind":"planFunctions","module":"dodal.plans"},{"kind":"planFunctions","module":"dodal.plan_stubs.wrapped"}]` | modules (must be installed in the venv) to fetch devices/plans from |
| worker.logging | object | `{"graylog":{"enabled":false,"url":"http://graylog-log-target.diamond.ac.uk:12232/"},"level":"INFO"}` | Configures logging. Port 12231 is the `dodal` input on graylog which will be renamed `blueapi` |
| worker.scratch | object | `{"repositories":[],"root":"/blueapi-plugins/scratch"}` | If initContainer is enabled the default branch of python projects in this section are installed into the venv *without their dependencies* |
| worker.stomp | object | `{"auth":{"password":"guest","username":"guest"},"enabled":false,"url":"http://rabbitmq:61613/"}` | Message bus configuration for returning status to GDA/forwarding documents downstream Password may be in the form ${ENV_VAR} to be fetched from an environment variable e.g. mounted from a SealedSecret |

----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to not have generated code in the repo or a way to ensure it stays up to date without someone having to remember to update it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linting will fail if the output should have changed, as it's run as a pre-commit hook

16 changes: 16 additions & 0 deletions helm/blueapi/README.md.gotmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{{ template "chart.header" . }}
{{ template "chart.deprecationWarning" . }}

{{ template "chart.description" . }}

{{ template "chart.homepageLine" . }}

{{ template "chart.maintainersSection" . }}

{{ template "chart.sourcesSection" . }}

{{ template "chart.requirementsSection" . }}

{{ template "chart.valuesSection" . }}

{{ template "helm-docs.versionFooter" . }}
25 changes: 24 additions & 1 deletion helm/blueapi/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,25 @@
1. Worker will be deployed with the following config:
Worker will be deployed with the following config:
{{- .Values.worker }}

1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
{{- range .paths }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
{{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "blueapi.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch its status by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "blueapi.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "blueapi.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "blueapi.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
15 changes: 15 additions & 0 deletions helm/blueapi/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,18 @@ data:
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: {{ .Values.tracing.otlp.protocol | default "http/protobuf" }}
OTEL_EXPORTER_OTLP_ENDPOINT: {{ required "OTLP export enabled but server address not set" .Values.tracing.otlp.server.host }}:{{ .Values.tracing.otlp.server.port | default 4318 }}
{{ end }}

---

{{- if .Values.initContainer.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "blueapi.fullname" . }}-init-config
data:
init_config.yaml: |-
scratch:
{{- toYaml .Values.worker.scratch | nindent 6 }}
{{- end }}

---
49 changes: 35 additions & 14 deletions helm/blueapi/templates/ingress.yaml
Original file line number Diff line number Diff line change
@@ -1,22 +1,43 @@
{{- if .Values.ingress.create -}}
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "blueapi.fullname" . }}
labels:
{{- include "blueapi.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
ingressClassName: nginx
{{- with .Values.ingress.className }}
ingressClassName: {{ . }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
- hosts:
- {{ required "A valid hostname must be provided" .Values.ingress.host }}
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
- host: {{ required "A valid hostname must be provided" .Values.ingress.host }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "blueapi.fullname" . }}
port:
number: {{ .Values.service.port }}
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
{{- with .pathType }}
pathType: {{ . }}
{{- end }}
backend:
service:
name: {{ include "blueapi.fullname" $ }}
port:
number: {{ $.Values.service.port }}
{{- end }}
{{- end }}
{{- end }}
11 changes: 0 additions & 11 deletions helm/blueapi/templates/init-configmap.yaml

This file was deleted.

12 changes: 7 additions & 5 deletions helm/blueapi/templates/service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@ apiVersion: v1
kind: Service
metadata:
name: {{ include "blueapi.fullname" . }}
labels:
{{- include "blueapi.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- name: http
port: {{ .Values.service.port }}
protocol: TCP
targetPort: http
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "blueapi.selectorLabels" . | nindent 4 }}
type: {{ .Values.service.type }}
1 change: 1 addition & 0 deletions helm/blueapi/templates/serviceaccount.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ metadata:
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
{{- end }}
53 changes: 39 additions & 14 deletions helm/blueapi/templates/statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ metadata:
labels:
{{- include "blueapi.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "blueapi.selectorLabels" . | nindent 6 }}
Expand All @@ -22,17 +21,28 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "blueapi.selectorLabels" . | nindent 8 }}
{{- include "blueapi.labels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
hostNetwork: {{ .Values.hostNetwork }}
{{- if .Values.hostNetwork }}
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
{{- end }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "blueapi.serviceAccountName" . }}
{{- with .Values.podSecurityContext }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
{{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
- name: worker-config
projected:
sources:
Expand All @@ -43,7 +53,7 @@ spec:
projected:
sources:
- configMap:
name: {{ include "blueapi.fullname" . }}-initconfig
name: {{ include "blueapi.fullname" . }}-init-config
- name: venv
emptyDir:
sizeLimit: 5Gi
Expand All @@ -65,19 +75,19 @@ spec:
- name: nslcd # Shared volume between main and sidecar container
emptyDir:
sizeLimit: 5Mi
{{- end }}
{{- end }}
{{- if .Values.initContainer.enabled }}
initContainers:
- name: setup-scratch
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{ ternary "-debug" "" .Values.debug.enabled }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
resources:
{{- .Values.initResources | default .Values.resources | toYaml | nindent 12 }}
command: [/bin/sh, -c]
command: ["/bin/sh", "-c"]
args:
- |
echo "Setting up scratch area"
blueapi -c /config/initconfig.yaml setup-scratch
blueapi -c /config/init_config.yaml setup-scratch
if [ $? -ne 0 ]; then echo 'Blueapi failed'; exit 1; fi;
echo "Exporting venv as artefact"
cp -r /venv/* /artefacts
Expand All @@ -95,12 +105,13 @@ spec:
mountPath: {{ .Values.worker.scratch.root }}
mountPropagation: HostToContainer
{{- end }}

{{- end }}
containers:
- name: {{ .Chart.Name }}
{{- with .Values.securityContext }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{ ternary "-debug" "" .Values.debug.enabled }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
Expand All @@ -116,12 +127,14 @@ spec:
containerPort: 8000
{{- end }}
protocol: TCP
{{- with .Values.resources }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: worker-config
mountPath: "/config"
readOnly: true
{{- with .Values.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if .Values.initContainer.enabled }}
{{- if .Values.initContainer.persistentVolume.enabled }}
- name: scratch
Expand All @@ -145,6 +158,18 @@ spec:
- "-c"
- "/config/config.yaml"
- "serve"
{{- with .Values.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.startupProbe }}
startupProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
envFrom:
- configMapRef:
Expand Down
Loading