Skip to content

Commit 5e3fff7

Browse files
PhilippMatthesSoWieMarkus
PhilippMatthes
and
SoWieMarkus
authored
Schedule commitments with new reservations operator & crd (#249)
## Change summary ### Reservations operator - Added syncer in operator that fetches commitments from limes and creates `ComputeReservation` resources for them. They are distinguished by `instance` or `bare` reservations. - Added reconciler in operator that takes all `instance` reservations and tries to place them on a host using the initial placement pipeline. If the placement succeeds, the reservation is considered `active`. - Added conf for supported hypervisor types, as we don't support the full filtering pipeline for vmware and ironic yet. `instance` reservations that don't match the hypervisor type are put into a terminal `failed` state. - Added logic that puts all `bare` reservations into a terminal `failed` state as we don't know yet how to handle these reservations and generate initial placement requests for them. - Added scaffolding with kubebuilder and separate dockerfile for operator, so we can reuse as many kubebuilder features as possible. ### Cortex core - Added kubernetes client to capacity filter so that reservations are considered. Note that the filter is currently sandboxed meaning it will have no effect for now until we de-sandbox it. - Added service account that binds to the appropriate reservations operator role, granting cortex core access to the reservations resources. - Refactored config loading so that we can reuse it in the reservations operator. - Added reservations operator api binds in go.mod so that cortex-core can access the v1alpha1 structs. ### CI/CD - Added image build workflow for cortex reservations operator - Added chart build workflow for cortex reservations operator - Run unit tests of reservations module and report coverage ### Tilt setup - Added reservations operator to the tilt setup. --------- Co-authored-by: Markus Wieland <[email protected]>
1 parent 4b20c88 commit 5e3fff7

File tree

99 files changed

+7580
-260
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+7580
-260
lines changed

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,5 @@
2121
/report.html
2222
/shell.nix
2323
/testing/
24+
25+
bin/

‎.github/workflows/push-charts.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,21 @@ jobs:
6262
CHART_PACKAGE=$(ls $CHART_DIR/*.tgz)
6363
helm push $CHART_PACKAGE oci://${{ env.REGISTRY }}/${{ github.repository }}/charts/
6464
done
65+
- name: Get all changed reservations Chart.yaml files
66+
id: changed-chart-yaml-files-reservations
67+
uses: tj-actions/changed-files@v46
68+
with:
69+
files: |
70+
reservations/dist/chart/Chart.yaml
71+
- name: Push reservations charts to registry
72+
if: steps.changed-chart-yaml-files-reservations.outputs.all_changed_files != ''
73+
shell: bash
74+
env:
75+
ALL_CHANGED_FILES: ${{ steps.changed-chart-yaml-files-reservations.outputs.all_changed_files }}
76+
run: |
77+
for CHART_FILE in ${ALL_CHANGED_FILES}; do
78+
CHART_DIR=$(dirname $CHART_FILE)
79+
helm package $CHART_DIR --dependency-update --destination $CHART_DIR
80+
CHART_PACKAGE=$(ls $CHART_DIR/*.tgz)
81+
helm push $CHART_PACKAGE oci://${{ env.REGISTRY }}/${{ github.repository }}/charts/
82+
done

‎.github/workflows/push-images.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,3 +98,44 @@ jobs:
9898
subject-name: ${{ env.REGISTRY }}/${{ github.repository }}
9999
subject-digest: ${{ steps.push_cortex.outputs.digest }}
100100
push-to-registry: true
101+
# Only build and push the reservations operator image if there are changes
102+
# in the reservations directory.
103+
- name: Get all changed reservations/ files
104+
id: changed_reservations_files
105+
uses: tj-actions/changed-files@v46
106+
with:
107+
files: |
108+
reservations/**
109+
- name: Docker Meta (Cortex Reservations)
110+
if: steps.changed_reservations_files.outputs.all_changed_files != ''
111+
id: meta_cortex_reservations
112+
uses: docker/metadata-action@v5
113+
with:
114+
images: ${{ env.REGISTRY }}/${{ github.repository }}-reservations-operator
115+
tags: |
116+
type=semver,pattern={{version}}
117+
type=semver,pattern={{major}}.{{minor}}
118+
type=sha
119+
latest
120+
- name: Build and Push Cortex Reservations Operator
121+
if: steps.changed_reservations_files.outputs.all_changed_files != ''
122+
id: push_cortex_reservations
123+
uses: docker/build-push-action@v6
124+
with:
125+
context: .
126+
file: Dockerfile.kubebuilder
127+
platforms: linux/amd64,linux/arm64
128+
push: true
129+
tags: ${{ steps.meta_cortex_reservations.outputs.tags }}
130+
labels: ${{ steps.meta_cortex_reservations.outputs.labels }}
131+
build-args: |
132+
GO_MOD_PATH=reservations
133+
GIT_TAG=${{ github.ref_name }}
134+
GIT_COMMIT=${{ github.sha }}
135+
- name: Generate Artifact Attestation for Cortex Reservations
136+
if: steps.changed_reservations_files.outputs.all_changed_files != ''
137+
uses: actions/attest-build-provenance@v2
138+
with:
139+
subject-name: ${{ env.REGISTRY }}/${{ github.repository }}-reservations-operator
140+
subject-digest: ${{ steps.push_cortex_reservations.outputs.digest }}
141+
push-to-registry: true

‎.github/workflows/test.yaml‎

Lines changed: 56 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,11 @@ jobs:
2020
with:
2121
go-version: 1.25.0
2222
- name: Test quickly without Docker
23-
run: go test -v ./...
23+
run: |
24+
echo "Testing main module..."
25+
go test -v ./...
26+
echo "Testing reservations module..."
27+
cd reservations && go test -v ./...
2428
2529
test-with-docker:
2630
# We don't need to run this longer test if the previous one already failed.
@@ -43,15 +47,27 @@ jobs:
4347
export GITHUB_ACTIONS=1
4448
export POSTGRES_CONTAINER=1
4549
export RABBITMQ_CONTAINER=1
50+
51+
echo "Running tests for main module..."
4652
go test -v \
4753
-coverpkg=./internal/... \
4854
-coverprofile=pr_profile.cov ./internal/...
4955
go tool cover -func pr_profile.cov > pr_func_coverage.txt
50-
- name: Upload coverage file
56+
57+
echo "Running tests for reservations module..."
58+
cd reservations
59+
go test -v \
60+
-coverpkg=./internal/... \
61+
-coverprofile=reservations_profile.cov ./internal/...
62+
go tool cover -func reservations_profile.cov > reservations_func_coverage.txt
63+
cd ..
64+
- name: Upload coverage files
5165
uses: actions/upload-artifact@v4
5266
with:
5367
name: pr-func-coverage
54-
path: pr_func_coverage.txt
68+
path: |
69+
pr_func_coverage.txt
70+
reservations/reservations_func_coverage.txt
5571
# Steps below are only executed if the workflow is triggered by a pull request
5672
- name: Delete old coverage comments (PR only)
5773
if: ${{ github.event_name == 'pull_request' }}
@@ -73,7 +89,7 @@ jobs:
7389
});
7490
}
7591
}
76-
- name: Download coverage file (PR only)
92+
- name: Download coverage files (PR only)
7793
if: ${{ github.event_name == 'pull_request' }}
7894
uses: actions/download-artifact@v5
7995
with:
@@ -87,20 +103,46 @@ jobs:
87103
const fs = require('fs');
88104
const path = require('path');
89105
90-
// Extract the last line of the coverage report
91-
const coverageReport = fs.readFileSync('pr_func_coverage.txt', 'utf8');
92-
const lines = coverageReport.trim().split('\n');
93-
const lastLine = lines[lines.length - 1];
94-
const coverageMatch = lastLine.match(/total:\s+\(statements\)\s+(\d+\.\d+)%/);
95-
const coveragePercentage = coverageMatch ? coverageMatch[1] : 'unknown';
106+
// Read main module coverage report
107+
const mainCoverageReport = fs.readFileSync('pr_func_coverage.txt', 'utf8');
108+
const mainLines = mainCoverageReport.trim().split('\n');
109+
const mainLastLine = mainLines[mainLines.length - 1];
110+
const mainCoverageMatch = mainLastLine.match(/total:\s+\(statements\)\s+(\d+\.\d+)%/);
111+
const mainCoveragePercentage = mainCoverageMatch ? mainCoverageMatch[1] : 'unknown';
112+
113+
// Read reservations module coverage report
114+
let reservationsCoverageReport = '';
115+
let reservationsCoveragePercentage = 'unknown';
116+
try {
117+
reservationsCoverageReport = fs.readFileSync('reservations/reservations_func_coverage.txt', 'utf8');
118+
const reservationsLines = reservationsCoverageReport.trim().split('\n');
119+
const reservationsLastLine = reservationsLines[reservationsLines.length - 1];
120+
const reservationsCoverageMatch = reservationsLastLine.match(/total:\s+\(statements\)\s+(\d+\.\d+)%/);
121+
reservationsCoveragePercentage = reservationsCoverageMatch ? reservationsCoverageMatch[1] : 'unknown';
122+
} catch (error) {
123+
reservationsCoverageReport = 'No coverage data available';
124+
}
96125
97126
let commentBody = '<!-- coverage-comment -->\n';
127+
commentBody += '## Test Coverage Report\n\n';
128+
129+
// Main module coverage
98130
commentBody += '<details>\n';
99-
commentBody += '<summary>Coverage in go module internal/: ';
100-
commentBody += coveragePercentage;
131+
commentBody += '<summary>Coverage in main module (internal/): ';
132+
commentBody += mainCoveragePercentage;
101133
commentBody += '%</summary>\n\n';
102134
commentBody += '```text\n';
103-
commentBody += coverageReport;
135+
commentBody += mainCoverageReport;
136+
commentBody += '```\n';
137+
commentBody += '</details>\n\n';
138+
139+
// Reservations module coverage
140+
commentBody += '<details>\n';
141+
commentBody += '<summary>Coverage in reservations module (reservations/internal/): ';
142+
commentBody += reservationsCoveragePercentage;
143+
commentBody += '%</summary>\n\n';
144+
commentBody += '```text\n';
145+
commentBody += reservationsCoverageReport;
104146
commentBody += '```\n';
105147
commentBody += '</details>\n';
106148
@@ -110,4 +152,4 @@ jobs:
110152
owner: context.repo.owner,
111153
repo: context.repo.repo,
112154
body: commentBody,
113-
});
155+
});

‎.github/workflows/update-appversion.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,17 @@ jobs:
5555
git add helm/library/cortex-postgres/Chart.yaml
5656
git commit -m "Bump cortex-postgres chart appVersions to ${{ steps.vars.outputs.sha }} [skip ci]" || echo "No changes to commit"
5757
git push origin HEAD:main
58+
59+
# Only bumped if there are changes in the reservations directory.
60+
- name: Update appVersion in cortex-reservations Chart.yaml
61+
if: steps.changed_reservations_files.outputs.all_changed_files != ''
62+
run: |
63+
sed -i 's/^\([ ]*appVersion:[ ]*\).*/\1"${{ steps.vars.outputs.sha }}"/' reservations/dist/chart/Chart.yaml
64+
- name: Commit and push changes for cortex-reservations
65+
if: steps.changed_reservations_files.outputs.all_changed_files != ''
66+
run: |
67+
git config user.name "github-actions[bot]"
68+
git config user.email "github-actions[bot]@users.noreply.github.com"
69+
git add reservations/dist/chart/Chart.yaml
70+
git commit -m "Bump cortex-reservations chart appVersions to ${{ steps.vars.outputs.sha }} [skip ci]" || echo "No changes to commit"
71+
git push origin HEAD:main

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ build/**
99
# Test binary, built with `go test -c`
1010
*.test
1111

12+
bin/
13+
1214
# Output of the go coverage tool, specifically when used with LiteIDE
1315
*.out
1416

‎Dockerfile.kubebuilder‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Build the manager binary
2+
FROM golang:1.24 AS builder
3+
ARG TARGETOS
4+
ARG TARGETARCH
5+
# Path of our go.mod
6+
ARG GO_MOD_PATH=.
7+
8+
WORKDIR /workspace
9+
# Copy shared cortex code
10+
COPY . /
11+
# Copy the Go Modules manifests
12+
COPY ${GO_MOD_PATH} .
13+
# cache deps before building and copying source so that we don't need to re-download as much
14+
# and so that source changes don't invalidate our downloaded layer
15+
RUN go mod download
16+
17+
# Build
18+
# the GOARCH has not a default value to allow the binary be built according to the host where the command
19+
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
20+
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
21+
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
22+
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
23+
24+
# Use distroless as minimal base image to package the manager binary
25+
# Refer to https://github.com/GoogleContainerTools/distroless for more details
26+
FROM gcr.io/distroless/static:nonroot
27+
WORKDIR /
28+
COPY --from=builder /workspace/manager .
29+
USER 65532:65532
30+
31+
ENTRYPOINT ["/manager"]

‎Tiltfile‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ if not os.getenv('TILT_VALUES_PATH'):
1111
fail("TILT_VALUES_PATH is not set.")
1212
if not os.path.exists(os.getenv('TILT_VALUES_PATH')):
1313
fail("TILT_VALUES_PATH "+ os.getenv('TILT_VALUES_PATH') + " does not exist.")
14+
tilt_values = os.getenv('TILT_VALUES_PATH')
1415

1516
# The upgrade job may take a long time to run, so it is disabled by default.
1617
enable_postgres_upgrade = False
@@ -22,6 +23,23 @@ helm_repo(
2223
labels=['Repositories'],
2324
)
2425

26+
def kubebuilder_binary_files(path):
27+
"""
28+
Return all usual binary files in a kubebuilder operator path.
29+
Can be used to perform selective watching on code paths for docker builds.
30+
"""
31+
return [path + '/cmd', path + '/api', path + '/internal', path + '/go.mod', path + '/go.sum']
32+
33+
########### Reservations Operator & CRDs
34+
docker_build('ghcr.io/cobaltcore-dev/cortex-reservations-operator', '.',
35+
dockerfile='Dockerfile.kubebuilder',
36+
build_args={'GO_MOD_PATH': 'reservations'},
37+
only=kubebuilder_binary_files('reservations') + ['internal/', 'go.mod', 'go.sum'],
38+
)
39+
local('sh helm/sync.sh reservations/dist/chart')
40+
k8s_yaml(helm('reservations/dist/chart', name='cortex-reservations', values=[tilt_values]))
41+
k8s_resource('reservations-controller-manager', labels=['Reservations'])
42+
2543
########### Dev Dependencies
2644
local('sh helm/sync.sh helm/dev/cortex-prometheus-operator')
2745
k8s_yaml(helm('./helm/dev/cortex-prometheus-operator', name='cortex-prometheus-operator')) # Operator
@@ -58,9 +76,9 @@ k8s_resource('cortex-plutono', port_forwards=[
5876
], labels=['Monitoring'])
5977

6078
########### Cortex Bundles
61-
tilt_values = os.getenv('TILT_VALUES_PATH')
6279
docker_build('ghcr.io/cobaltcore-dev/cortex', '.', only=[
6380
'internal/', 'commands/', 'main.go', 'go.mod', 'go.sum', 'Makefile',
81+
'reservations/api/', # API module of the reservations operator needed for the scheduler.
6482
])
6583
docker_build('ghcr.io/cobaltcore-dev/cortex-postgres', 'postgres')
6684

‎cortex.secrets.example.yaml‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,21 @@ sharedSSOCert: &sharedSSOCert
2020
# If true, the certificate is not verified.
2121
selfSigned: false
2222

23+
# Shared keystone credentials to use.
24+
keystone: &keystone
25+
url: https://path-to-keystone/v3
26+
sso: *sharedSSOCert
27+
username: openstack-user-with-all-project-read-access
28+
password: openstack-user-password
29+
projectName: openstack-project-of-user
30+
userDomainName: openstack-domain-of-user
31+
projectDomainName: openstack-domain-of-project-scoped-to
32+
33+
# Custom configuration for the reservations operator.
34+
reservations:
35+
secrets:
36+
keystone: *keystone
37+
2338
# These values will be shared across cortex-nova, cortex-manila, ... locally.
2439
cortex-core:
2540
secrets:
@@ -42,11 +57,4 @@ cortex-core:
4257
sso: *sharedSSOCert
4358
provides: [netapp_aggregate_labels_metric, netapp_node_metric]
4459
# Override the endpoints and credentials to your OpenStack.
45-
keystone:
46-
url: https://path-to-keystone/v3
47-
sso: *sharedSSOCert
48-
username: openstack-user-with-all-project-read-access
49-
password: openstack-user-password
50-
projectName: openstack-project-of-user
51-
userDomainName: openstack-domain-of-user
52-
projectDomainName: openstack-domain-of-project-scoped-to
60+
keystone: *keystone

0 commit comments

Comments
 (0)