Skip to content

Commit f99140e

Browse files
cluxclux
authored
Run e2e,integration tests against supported kubernetes versions (kube-rs#924)
* Run e2e tests against supported kubernetes versions As part of Kubernetes version policies proposed in kube-rs/website#19 for kube-rs#684 Signed-off-by: clux <[email protected]> * add integration tests as specific job to run it against mk8sv Signed-off-by: clux <[email protected]> * rename tests and try to fix integration Signed-off-by: clux <[email protected]> * fix k3d args? Signed-off-by: clux <[email protected]> * fix e2e Signed-off-by: clux <[email protected]> * test against v1.19 Signed-off-by: clux <[email protected]> * badge update Signed-off-by: clux <[email protected]> * s/MINK8SV/MK8SV Signed-off-by: clux <[email protected]> * Big e2e rework (WIP) Now two e2e tests, and more feature combinations. Signed-off-by: clux <[email protected]> * avoid rustls local due to the issues Signed-off-by: clux <[email protected]> * split e2e tests into own file and clean up when to matrix Signed-off-by: clux <[email protected]> * running bump-k8s also bumps mk8sv Signed-off-by: clux <[email protected]> * avoid illegal anchors and add badge bumping Signed-off-by: clux <[email protected]> * better version checks for mk8sv and msrv on ci Signed-off-by: clux <[email protected]> * move some ci around for consistency in naming Signed-off-by: clux <[email protected]> * stray command Signed-off-by: clux <[email protected]> * fix syntax errors? Signed-off-by: clux <[email protected]> * fix test warning from kube-derive after apiext removal Signed-off-by: clux <[email protected]> * evars from tests Signed-off-by: clux <[email protected]> * better run description on integration jobs Signed-off-by: clux <[email protected]> * better output for flaky integration test Signed-off-by: clux <[email protected]> * fmt + give test more time? Signed-off-by: clux <[email protected]> * sanity after timeout. seems likely cause. Signed-off-by: clux <[email protected]> * no need for an agent in k3d cmd Signed-off-by: clux <[email protected]> * how slow is this actions cluster? Signed-off-by: clux <[email protected]> * re-trigger once more to check * apiserver, are you ok? Signed-off-by: clux <[email protected]> * another empty commit to retrigger Signed-off-by: clux <[email protected]> * remove duplicate changes from kube-rs#924 (won't build until that is merged) Signed-off-by: clux <[email protected]>
1 parent 4bb19fb commit f99140e

18 files changed

+363
-200
lines changed

‎.github/workflows/ci.yml

+147-48
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
1-
name: CI
1+
name: test
22
on:
33
pull_request:
44

5+
env:
6+
RUST_BACKTRACE: 1
7+
58
jobs:
6-
cargo-test:
9+
unit:
710
strategy:
811
# Prevent GitHub from cancelling all in-progress jobs when a matrix job fails.
912
fail-fast: false
@@ -66,7 +69,7 @@ jobs:
6669
run: cargo test -p kube-examples --example crd_derive_no_schema --no-default-features --features=openssl-tls,latest
6770
if: matrix.os == 'ubuntu-latest'
6871

69-
check-msrv:
72+
msrv:
7073
# Run `cargo check` on our minimum supported Rust version
7174
runs-on: ubuntu-latest
7275
steps:
@@ -89,6 +92,18 @@ jobs:
8992
command: check
9093
args: --all
9194

95+
- name: Check rust-version keys matches MSRV consistently
96+
run: |
97+
if [[ $(cat ./*/Cargo.toml | grep "rust-version" | uniq | wc -l) -gt 1 ]]; then
98+
echo "inconsistent rust-version keys set in various kube-crates:"
99+
rg "rust-version" ./*/Cargo.toml
100+
exit 1
101+
fi
102+
if ! cat kube/Cargo.toml | grep "rust-version" | grep "${{ steps.msrv.outputs.msrv }}"; then
103+
echo "msrv policy inconsistent with rust-version key"
104+
exit 1
105+
fi
106+
92107
- name: Check devcontainer matches MSRV
93108
run: |
94109
versions=$(sed -nE 's|^FROM (.*/)?rust:([^ ]+)|\2|p' .devcontainer/Dockerfile)
@@ -100,86 +115,170 @@ jobs:
100115
exit 1
101116
fi
102117
103-
e2e:
104-
# e2e tests are docker on linux
118+
integration:
105119
runs-on: ubuntu-latest
120+
strategy:
121+
# Prevent GitHub from cancelling all in-progress jobs when a matrix job fails.
122+
fail-fast: false
123+
matrix:
124+
# Run these tests against older clusters as well
125+
k8s: [v1.19, latest]
106126
steps:
107127
- uses: actions/checkout@v2
128+
- uses: actions-rs/toolchain@v1
129+
with:
130+
override: true
131+
toolchain: stable
132+
profile: minimal
133+
# Smart caching for Rust projects.
134+
# Includes workaround for macos cache corruption.
135+
# - https://github.com/rust-lang/cargo/issues/8603
136+
# - https://github.com/actions/cache/issues/403
137+
- uses: Swatinem/rust-cache@v1
138+
139+
- uses: nolar/setup-k3d-k3s@v1
140+
with:
141+
version: ${{matrix.k8s}}
142+
# k3d-kube
143+
k3d-name: kube
144+
# Used to avoid rate limits when fetching the releases from k3s repo.
145+
# Anonymous access is limited to 60 requests / hour / worker
146+
# github-token: ${{ secrets.GITHUB_TOKEN }}
147+
k3d-args: "--no-lb --k3s-arg --no-deploy=traefik,servicelb,metrics-server@server:*"
148+
149+
# Real CI work starts here
150+
- name: Build workspace
151+
run: cargo build
152+
153+
# Run the equivalent of `just integration`
154+
- name: Run all default features integration library tests
155+
run: cargo test --lib --all -- --ignored
156+
- name: Run all facade integration library tests with extra features
157+
run: cargo test -p kube --lib --features=derive,runtime -- --ignored --nocapture
158+
- name: Run all client integration library tests with rustls and ws
159+
run: cargo test -p kube-client --lib --features=rustls-tls,ws -- --ignored
160+
- name: Run derive example tests
161+
run: cargo run -p kube-examples --example crd_derive
162+
- name: Run crd example tests
163+
run: cargo run -p kube-examples --example crd_api
164+
165+
166+
mk8sv:
167+
# comile check e2e tests against mk8sv
168+
runs-on: ubuntu-latest
169+
steps:
170+
- uses: actions/checkout@v2
171+
172+
- name: Find MK8SV
173+
id: mk8sv
174+
run: |
175+
MK8SV=$(grep MK8SV README.md | grep -oE "[[:digit:]]+\.[[:digit:]]+" | head -n 1)
176+
echo $MK8SV
177+
echo ::set-output name=mk8sv::${MK8SV}
178+
echo ::set-output name=mk8svdash::v${MK8SV/\./_}
179+
180+
- name: Check ci jobs run against advertised MK8SV
181+
run: |
182+
if ! grep "${{ steps.mk8sv.outputs.mk8sv }}" -q .github/workflows/ci.yml; then
183+
echo "mk8sv not set correctly in tests"
184+
exit 1
185+
fi
186+
if ! grep "${{ steps.mk8sv.outputs.mk8svdash }}" e2e/Cargo.toml | grep mk8sv; then
187+
echo "mk8sv not set correctly in e2e features"
188+
exit 1
189+
fi
190+
191+
- uses: actions-rs/toolchain@v1
192+
with:
193+
override: true
194+
toolchain: stable
195+
profile: minimal
196+
# Smart caching for Rust projects.
197+
# Includes workaround for macos cache corruption.
198+
# - https://github.com/rust-lang/cargo/issues/8603
199+
# - https://github.com/actions/cache/issues/403
200+
- uses: Swatinem/rust-cache@v1
108201

202+
- uses: nolar/setup-k3d-k3s@v1
203+
with:
204+
version: v1.19
205+
# k3d-kube
206+
k3d-name: kube
207+
# Used to avoid rate limits when fetching the releases from k3s repo.
208+
# Anonymous access is limited to 60 requests / hour / worker
209+
# github-token: ${{ secrets.GITHUB_TOKEN }}
210+
k3d-args: "--no-lb --k3s-arg --no-deploy=traefik,servicelb,metrics-server@server:*"
211+
212+
# Real CI work starts here
213+
- name: Build workspace
214+
run: cargo build
215+
216+
- name: boot openssl, latest k8s
217+
run: cargo run -p e2e --bin boot --features=openssl,latest
218+
- name: boot openssl, minimum supported k8s
219+
run: cargo run -p e2e --bin boot --features=openssl,mk8sv
220+
# no rustls local yet: https://github.com/kube-rs/kube-rs/issues/153
221+
# we could edit kubeconfig and /etc/hosts to have the following lines:
222+
# /etc/hosts: 127.0.0.1 k3d.local
223+
# kubeconfig: server: https://k3d.local:36729 (e.g. swap out 0.0.0.0 under cluster)
224+
# Together, that makes it work locally. Maybe that can be done with some k3d-args to host-aliases.
225+
#- name: boot rustls, latest k8s
226+
# run: cargo run -p e2e --bin boot --features=rustls,latest
227+
#- name: boot rustls, minimum supported k8s
228+
# run: cargo run -p e2e --bin boot --features=rustls,mk8sv
229+
230+
in-cluster:
231+
# in-cluster e2e via docker on linux
232+
runs-on: ubuntu-latest
233+
strategy:
234+
# Prevent GitHub from cancelling all in-progress jobs when a matrix job fails.
235+
fail-fast: false
236+
matrix:
237+
tls: [openssl, rustls]
238+
steps:
239+
- uses: actions/checkout@v2
109240
- uses: actions/cache@v2
110241
with:
111242
path: |
112243
~/.cargo/registry/index
113244
~/.cargo/registry/cache
114245
~/.cargo/git
115246
target
116-
key: musl-cargo-${{ hashFiles('**/Cargo.toml') }}
247+
key: musl-cargo-${{ hashFiles('**/Cargo.toml') }}-${{matrix.tls}}
117248

118249
- uses: nolar/setup-k3d-k3s@v1
119250
with:
120-
version: v1.20
251+
version: latest
121252
# k3d-kube
122253
k3d-name: kube
123254
# Used to avoid rate limits when fetching the releases from k3s repo.
124255
# Anonymous access is limited to 60 requests / hour / worker
125256
# github-token: ${{ secrets.GITHUB_TOKEN }}
257+
k3d-args: "--no-lb --k3s-arg --no-deploy=traefik,servicelb,metrics-server@server:*"
126258

127-
- name: Compile dapp
259+
- name: Compile e2e job against ${{matrix.tls}}
128260
run: |
129261
mkdir -p ~/.cargo/{git,registry}
130262
docker run --rm -t \
131263
--mount type=bind,source=${{ github.workspace }},target=/volume \
132264
--mount type=bind,source=$HOME/.cargo/registry,target=/root/.cargo/registry \
133265
--mount type=bind,source=$HOME/.cargo/git,target=/root/.cargo/git \
134266
clux/muslrust:stable \
135-
cargo build -p e2e --release -v
136-
cp target/x86_64-unknown-linux-musl/release/dapp e2e/
267+
cargo build -p e2e --release --bin=job --features=latest,${{matrix.tls}} -v
268+
cp target/x86_64-unknown-linux-musl/release/job e2e/
137269
138270
- name: Build image
139-
run: "docker build -t clux/kube-dapp:${{ github.sha }} e2e/"
271+
run: "docker build -t clux/kube-e2e:${{ github.sha }} e2e/"
140272
- name: Import image
141-
run: "k3d image import clux/kube-dapp:${{ github.sha }} --cluster kube"
273+
run: "k3d image import clux/kube-e2e:${{ github.sha }} --cluster kube"
142274
- run: sed -i 's/latest/${{ github.sha }}/g' e2e/deployment.yaml
143275

144276
- name: Create resource
145277
run: kubectl apply -f e2e/deployment.yaml -n apps
146278
- run: kubectl get all -n apps
147-
- run: kubectl describe jobs/dapp -n apps
279+
- run: kubectl describe jobs/e2e -n apps
148280
- name: Wait for job to complete
149281
run: |
150-
kubectl wait --for=condition=complete job/dapp -n apps --timeout=50s || kubectl logs -f job/dapp -n apps
282+
kubectl wait --for=condition=complete job/e2e -n apps --timeout=50s || kubectl logs -f job/e2e -n apps
151283
kubectl get all -n apps
152-
kubectl wait --for=condition=complete job/dapp -n apps --timeout=10s || kubectl get pods -n apps | grep dapp | grep Completed
153-
154-
cargo-deny:
155-
name: Run cargo deny
156-
runs-on: ubuntu-latest
157-
strategy:
158-
matrix:
159-
checks:
160-
- advisories
161-
- bans licenses sources
162-
163-
# Prevent sudden announcement of a new advisory from failing ci:
164-
continue-on-error: ${{ matrix.checks == 'advisories' }}
165-
166-
steps:
167-
- uses: actions/checkout@v2
168-
- uses: EmbarkStudios/cargo-deny-action@v1
169-
with:
170-
command: check ${{ matrix.checks }}
171-
172-
rustfmt:
173-
name: Run rustfmt
174-
runs-on: ubuntu-latest
175-
steps:
176-
- uses: actions/checkout@v2
177-
- uses: actions-rs/[email protected]
178-
with:
179-
profile: minimal
180-
toolchain: nightly
181-
components: rustfmt
182-
override: true
183-
- name: Run rustfmt
184-
id: rustfmt
185-
run: rustfmt +nightly --edition 2018 --check $(find . -type f -iname *.rs)
284+
kubectl wait --for=condition=complete job/e2e -n apps --timeout=10s || kubectl get pods -n apps | grep e2e | grep Completed

‎.github/workflows/coverage.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Coverage
1+
name: coverage
22

33
on:
44
push:
@@ -7,7 +7,7 @@ on:
77
pull_request:
88

99
jobs:
10-
rust:
10+
tarpaulin-codecov:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- name: Checkout

‎.github/workflows/devcontainer.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Devcontainer
1+
name: devcontainer
22

33
# When a pull request is opened that changes the Devcontainer configuration,
44
# ensure that the container continues to build properly.

‎.github/workflows/lint.yml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: lint
2+
on:
3+
pull_request:
4+
5+
jobs:
6+
cargo-deny:
7+
name: cargo deny
8+
runs-on: ubuntu-latest
9+
strategy:
10+
matrix:
11+
checks:
12+
- advisories
13+
- bans licenses sources
14+
15+
# Prevent sudden announcement of a new advisory from failing ci:
16+
continue-on-error: ${{ matrix.checks == 'advisories' }}
17+
18+
steps:
19+
- uses: actions/checkout@v2
20+
- uses: EmbarkStudios/cargo-deny-action@v1
21+
with:
22+
command: check ${{ matrix.checks }}
23+
24+
rustfmt:
25+
name: rustfmt
26+
runs-on: ubuntu-latest
27+
steps:
28+
- uses: actions/checkout@v2
29+
- uses: actions-rs/[email protected]
30+
with:
31+
profile: minimal
32+
toolchain: nightly
33+
components: rustfmt
34+
override: true
35+
- name: Run rustfmt
36+
id: rustfmt
37+
run: rustfmt +nightly --edition 2018 --check $(find . -type f -iname *.rs)

‎.github/workflows/release.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Release
1+
name: release
22

33
on:
44
push:
@@ -8,6 +8,7 @@ on:
88
jobs:
99
build:
1010
runs-on: ubuntu-latest
11+
if: ${{ github.repository_owner == 'kube-rs' }}
1112
steps:
1213
- name: Checkout
1314
uses: actions/checkout@v2

‎.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
/target
44
**/*.rs.bk
55
Cargo.lock
6-
integration/dapp
6+
e2e/boot
7+
e2e/job
78
examples/admission-controller-tls*
89
examples/admission_extfile.cnf
910
examples/ca.*

‎README.md

+3-4
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,9 @@
22

33
[![Crates.io](https://img.shields.io/crates/v/kube.svg)](https://crates.io/crates/kube)
44
[![Rust 1.60](https://img.shields.io/badge/MSRV-1.60-dea584.svg)](https://github.com/rust-lang/rust/releases/tag/1.60.0)
5-
[![Discord chat](https://img.shields.io/discord/500028886025895936.svg?logo=discord&style=plastic)](https://discord.gg/tokio)
6-
[![Client Capabilities](https://img.shields.io/badge/Kubernetes%20client-Silver-blue.svg?style=plastic&colorB=C0C0C0&colorA=306CE8)](https://github.com/kubernetes/design-proposals-archive/blob/main/api-machinery/csi-new-client-library-procedure.md#client-capabilities)
7-
[![Client Support Level](https://img.shields.io/badge/kubernetes%20client-beta-green.svg?style=plastic&colorA=306CE8)](https://github.com/kubernetes/design-proposals-archive/blob/main/api-machinery/csi-new-client-library-procedure.md#client-support-level)
5+
[![Tested against Kubernetes 1.19 and above](https://img.shields.io/badge/MK8SV-1.19-326ce5.svg)](https://kube.rs/kubernetes-version)
86
[![Best Practices](https://bestpractices.coreinfrastructure.org/projects/5413/badge)](https://bestpractices.coreinfrastructure.org/projects/5413)
7+
[![Discord chat](https://img.shields.io/discord/500028886025895936.svg?logo=discord&style=plastic)](https://discord.gg/tokio)
98

109
A [Rust](https://rust-lang.org/) client for [Kubernetes](http://kubernetes.io) in the style of a more generic [client-go](https://github.com/kubernetes/client-go), a runtime abstraction inspired by [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime), and a derive macro for [CRDs](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) inspired by [kubebuilder](https://book.kubebuilder.io/reference/generating-crd.html). Hosted by [CNCF](https://cncf.io/) as a [Sandbox Project](https://www.cncf.io/sandbox-projects/)
1110

@@ -149,7 +148,7 @@ Here `reconcile` and `error_policy` refer to functions you define. The first wil
149148

150149
## Rustls
151150

152-
Kube has basic support ([with caveats](https://github.com/kube-rs/kube-rs/issues?q=is%3Aissue+is%3Aopen+rustls)) for [rustls](https://github.com/ctz/rustls) as a replacement for the `openssl` dependency. To use this, turn off default features, and enable `rustls-tls`:
151+
Kube has basic support ([with caveats](https://github.com/kube-rs/kube-rs/issues?q=is%3Aopen+is%3Aissue+label%3Arustls)) for [rustls](https://github.com/ctz/rustls) as a replacement for the `openssl` dependency. To use this, turn off default features, and enable `rustls-tls`:
153152

154153
```toml
155154
[dependencies]

‎e2e/Cargo.toml

+16-6
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,25 @@ license = "Apache-2.0"
1010
release = false
1111

1212
[[bin]]
13-
name = "dapp"
14-
path = "dapp.rs"
13+
name = "job"
14+
path = "job.rs"
15+
16+
[[bin]]
17+
name = "boot"
18+
path = "boot.rs"
19+
20+
[features]
21+
latest = ["k8s-openapi/v1_24"]
22+
mk8sv = ["k8s-openapi/v1_19"]
23+
rustls = ["kube/rustls-tls"]
24+
openssl = ["kube/openssl-tls"]
1525

1626
[dependencies]
1727
anyhow = "1.0.44"
18-
env_logger = "0.9.0"
28+
tracing = "0.1.29"
29+
tracing-subscriber = "0.3.3"
1930
futures = "0.3.17"
20-
kube = { path = "../kube", version = "^0.73.0", default-features = false, features = ["client", "rustls-tls"] }
21-
k8s-openapi = { version = "0.15.0", features = ["v1_24"], default-features = false }
22-
log = "0.4.11"
31+
kube = { path = "../kube", version = "^0.73.0", default-features = false, features = ["client", "runtime", "ws", "admission", "gzip"] }
32+
k8s-openapi = { version = "0.15.0", default-features = false }
2333
serde_json = "1.0.68"
2434
tokio = { version = "1.14.0", features = ["full"] }

0 commit comments

Comments
 (0)