test: add compatibility versions feature gate test as a nightly prow periodic job #34257

aaron-prindle · 2025-02-03T18:36:31Z

This PR adds a feature get test to the compatibility version prow periodic jobs. This test is adapted from the closed PR here: kubernetes/kubernetes#128138 which attempted a similar test upstream (it was decided test-infra was the better place for the test in the discussion there). This feature gate test builds and runs the latest k8s, hits the /metrics endpoint fetching the current feature gates and their information (stage, lockToDefault, enabled/disabled). Then it pulls the n-1 k8s repo and grabs the versioned_feature_list.yaml and unversioned_feature_list.yaml files from that repo which are the source of truth for n-1 feature gates and compares them making sure that the feature gates for n vs n-1 fall within the necessary guarantees for the compatibility versions feature.

Example test run by running the script locally w/ same env var config as the prow test:
https://gist.github.com/aaprindle/254f748d008afc11063ba3479f7238ff

aaron-prindle · 2025-02-03T18:37:13Z

/assign @BenTheElder

aaron-prindle · 2025-02-03T20:54:16Z

/cc @Jefftree

Jefftree

General question about test-infra changes: Is it possible to see a sample run of this before merging or is that not possible?

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

aaron-prindle · 2025-02-03T21:35:45Z

General question about test-infra changes: Is it possible to see a sample run of this before merging or is that not possible?

I don't believe it is possible, I have been testing the scripts manually. I have attached a gist of a test output to the PR description.

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

BenTheElder · 2025-02-03T23:07:46Z

General question about test-infra changes: Is it possible to see a sample run of this before merging or is that not possible?

possible, but a big pain (you have to setup everything yourself, see e.g. pj-on-kind.sh, which is going to have problems with e2e tests)

... and prow has ~no investment right now, just barely community-provided KTLO

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

BenTheElder · 2025-02-03T23:10:51Z

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

+      - wrapper.sh
+      - bash
+      - -c
+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ./../test-infra/experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh


xref: #33980

Apologies is the recommendation here related to?
#33594

From reading over #33980 I'm a bit confused what the actionable change would be to make to the script here is, can you elaborate a bit on how I might use the precompiled binaries here? Thanks!

Sorry -- I'm just trying to track these for later, it's not a blocker.

Basically you use kind build node-image with one of the CI builds of Kubernetes instead of from source.
We will also have to make sure the commit from that build gets recorded to testgrid (I forget the specifics of how that works), if / when we convert one of the scripts we can apply it in multiple places.

But it's not a blocker.

Ah makes sense, thanks. I'll track this work through #33594, added a reference there to #33980. I'll work on a separate PR converting all of the compatibility-versions pieces to use prebuilt binaries when I tackle #33594

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

BenTheElder · 2025-02-03T23:19:58Z

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

+  kubectl get --raw /metrics > "${output_file}"
+}
+
+validate_feature_gates() {


Seems like it would make sense to factor out this part into a test script independent of all the e2e setup bits, at some point we may more directly share the rest? Also it seems like most (all?) of this [test] would work on other clusters?

I've refactored validate_feature_gates into its own script now:
experiment/compatibility-versions/validate-compatibility-versions-feature-gates.sh

which is called from the main/entrypoint script:
experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

Thanks! I could see us hoisting this out or even rewriting it in go later for use by more jobs (maybe into k/k and run against GCE clusters as well?) :-)

BenTheElder · 2025-02-05T18:48:46Z

/sig api-machinery

BenTheElder · 2025-02-05T22:29:43Z

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

+runtimeConfig: ${runtime_config}
+kubeadmConfigPatches:
+- |
+  kind: ClusterConfiguration


We're also going to have to start targeting these to specific versions and add a kubeadm v1alpha4 patch soon: kubernetes-sigs/kind#3847

Going to be a headache, at least trying to track the places in-project where we're running kind @ HEAD

BenTheElder · 2025-02-05T22:32:56Z

experiment/compatibility-versions/validate-compatibility-versions-feature-gates.sh

+  touch "${results_file}"
+
+  # Parse /metrics -> actual_features[featureName] = 0 or 1
+  declare -A actual_features


worth noting: when we have a lot of arrays that aren't something trivial like exec args that's a sign that we probably want to reach for a more structured language

+1, I originally wrote this test as an integration test for k8s/k8s in Golang but ported it to bash for use in test-infra after some discussion on the original PR:
kubernetes/kubernetes#128138

ACK, I commented there to continue the conversation 👍

(and I think for now bash is reasonable, but something to be wary of longterm, especially if we know we're going to need more functionality)

BenTheElder · 2025-02-05T22:38:04Z

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh

@@ -0,0 +1,259 @@
+#!/bin/bash


this can be a follow-up but we usually use #!/usr/bin/env bash, this guarantees that we use the preferred copy in PATH and is a little more portable (/usr/bin/env is virtually guaranteed due to env being POSIX and so many scripts doing this, but bash may not be in /bin, e.g. on my work mac I have installed newer bash via homebrew and kubernetes scripts may require that)

same goes for the other script.

https://en.wikipedia.org/wiki/Shebang_(Unix)#Portability

Thanks for flagging this, I've updated all scripts to use:

#!/usr/bin/env bash

BenTheElder · 2025-02-05T22:39:16Z

experiment/compatibility-versions/validate-compatibility-versions-feature-gates.sh

+  done
+}
+
+# --- Main execution when script is called directly ---


I would recommend just only supporting calling and condense this down to a single codepath / exec it from the other script instead of trying to support both exec and sourcing.

(up to you, but I'd suggest considering this before it gets wider usage, feel free to /hold cancel and punt)

Ah makes sense, I've updated the script to support only being exec'd for now.

BenTheElder

/lgtm
/approve

BenTheElder · 2025-02-05T22:40:19Z

/hold

k8s-ci-robot · 2025-02-05T22:40:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aaron-prindle, BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [BenTheElder]
~~experiment/compatibility-versions/OWNERS~~ [BenTheElder,aaron-prindle]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…periodic job

aaron-prindle · 2025-02-05T23:20:25Z

/hold cancel

BenTheElder

/lgtm

k8s-ci-robot · 2025-02-05T23:34:21Z

@aaron-prindle: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key compatibility-versions-e2e.yaml using file config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml

In response to this:

This PR adds a feature get test to the compatibility version prow periodic jobs. This test is adapted from the closed PR here: kubernetes/kubernetes#128138 which attempted a similar test upstream (it was decided test-infra was the better place for the test in the discussion there). This feature gate test builds and runs the latest k8s, hits the /metrics endpoint fetching the current feature gates and their information (stage, lockToDefault, enabled/disabled). Then it pulls the n-1 k8s repo and grabs the versioned_feature_list.yaml and unversioned_feature_list.yaml files from that repo which are the source of truth for n-1 feature gates and compares them making sure that the feature gates for n vs n-1 fall within the necessary guarantees for the compatibility versions feature.

Example test run by running the script locally w/ same env var config as the prow test:
https://gist.github.com/aaprindle/254f748d008afc11063ba3479f7238ff

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 3, 2025

k8s-ci-robot requested review from MadhavJivrajani and wojtek-t February 3, 2025 18:36

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. area/config Issues or PRs related to code in /config area/jobs sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Feb 3, 2025

k8s-ci-robot assigned BenTheElder Feb 3, 2025

aaron-prindle force-pushed the compat-versions-ft-gate-test branch 2 times, most recently from 05e2216 to 3d5d4f7 Compare February 3, 2025 19:12

k8s-ci-robot requested a review from Jefftree February 3, 2025 20:54

aaron-prindle force-pushed the compat-versions-ft-gate-test branch from 3d5d4f7 to 7c7effc Compare February 3, 2025 20:56

Jefftree reviewed Feb 3, 2025

View reviewed changes

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml Outdated Show resolved Hide resolved

aaron-prindle force-pushed the compat-versions-ft-gate-test branch from 7c7effc to 28998c5 Compare February 3, 2025 21:27

aaron-prindle force-pushed the compat-versions-ft-gate-test branch 2 times, most recently from 35f6a6c to b79b493 Compare February 3, 2025 22:33

Jefftree reviewed Feb 3, 2025

View reviewed changes

config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml Outdated Show resolved Hide resolved

experiment/compatibility-versions/compatibility-versions-feature-gate-test.sh Outdated Show resolved Hide resolved

BenTheElder reviewed Feb 3, 2025

View reviewed changes

aaron-prindle force-pushed the compat-versions-ft-gate-test branch 6 times, most recently from 182200d to a7cca0e Compare February 5, 2025 17:47

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Feb 5, 2025

BenTheElder reviewed Feb 5, 2025

View reviewed changes

aaron-prindle force-pushed the compat-versions-ft-gate-test branch from a7cca0e to 0ebca9f Compare February 5, 2025 22:39

BenTheElder reviewed Feb 5, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 5, 2025

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Feb 5, 2025

test: add compatibility versions feature gate test as a nightly prow …

2220b0b

…periodic job

aaron-prindle force-pushed the compat-versions-ft-gate-test branch from 0ebca9f to 2220b0b Compare February 5, 2025 23:20

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 5, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 5, 2025

BenTheElder reviewed Feb 5, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 5, 2025

k8s-ci-robot merged commit f414bfa into kubernetes:master Feb 5, 2025
6 of 7 checks passed

aaron-prindle mentioned this pull request Feb 10, 2025

K8s Compatibility Versions (--emulated-version) feature should have integration test to validate feature gate compatibility kubernetes/kubernetes#127947

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add compatibility versions feature gate test as a nightly prow periodic job #34257

test: add compatibility versions feature gate test as a nightly prow periodic job #34257

aaron-prindle commented Feb 3, 2025 •

edited

Loading

aaron-prindle commented Feb 3, 2025

aaron-prindle commented Feb 3, 2025

Jefftree left a comment

aaron-prindle commented Feb 3, 2025

BenTheElder commented Feb 3, 2025

BenTheElder Feb 3, 2025

aaron-prindle Feb 5, 2025 •

edited

Loading

BenTheElder Feb 5, 2025

aaron-prindle Feb 5, 2025

BenTheElder Feb 3, 2025 •

edited

Loading

aaron-prindle Feb 5, 2025

BenTheElder Feb 5, 2025

BenTheElder commented Feb 5, 2025

BenTheElder Feb 5, 2025

BenTheElder Feb 5, 2025

aaron-prindle Feb 5, 2025 •

edited

Loading

BenTheElder Feb 5, 2025

BenTheElder Feb 5, 2025

BenTheElder Feb 5, 2025

aaron-prindle Feb 5, 2025

BenTheElder Feb 5, 2025

BenTheElder Feb 5, 2025

aaron-prindle Feb 5, 2025

BenTheElder left a comment

BenTheElder commented Feb 5, 2025

k8s-ci-robot commented Feb 5, 2025

aaron-prindle commented Feb 5, 2025

BenTheElder left a comment

k8s-ci-robot commented Feb 5, 2025

test: add compatibility versions feature gate test as a nightly prow periodic job #34257

test: add compatibility versions feature gate test as a nightly prow periodic job #34257

Conversation

aaron-prindle commented Feb 3, 2025 • edited Loading

aaron-prindle commented Feb 3, 2025

aaron-prindle commented Feb 3, 2025

Jefftree left a comment

Choose a reason for hiding this comment

aaron-prindle commented Feb 3, 2025

BenTheElder commented Feb 3, 2025

Choose a reason for hiding this comment

aaron-prindle Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented Feb 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder left a comment

Choose a reason for hiding this comment

BenTheElder commented Feb 5, 2025

k8s-ci-robot commented Feb 5, 2025

aaron-prindle commented Feb 5, 2025

BenTheElder left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 5, 2025

aaron-prindle commented Feb 3, 2025 •

edited

Loading

aaron-prindle Feb 5, 2025 •

edited

Loading

BenTheElder Feb 3, 2025 •

edited

Loading

aaron-prindle Feb 5, 2025 •

edited

Loading