Skip to content

Commit 737507b

Browse files
stefanprodanstefanprodan
authored
Merge pull request #37 from stefanprodan/track-configs
 Track changes in ConfigMaps and Secrets
2 parents b8a7ea8 + 4bcf82d commit 737507b

18 files changed

+1453
-628
lines changed

‎README.md

+12-4
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ ClusterIP [services](https://kubernetes.io/docs/concepts/services-networking/ser
3838
Istio [virtual services](https://istio.io/docs/reference/config/istio.networking.v1alpha3/#VirtualService))
3939
to drive the canary analysis and promotion.
4040

41+
Flagger keeps track of ConfigMaps and Secrets referenced by a Kubernetes Deployment and triggers a canary analysis if any of those objects change.
42+
When promoting a workload in production, both code (container images) and configuration (config maps and secrets) are being synchronised.
43+
4144
![flagger-overview](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-overview.png)
4245

4346
Gated canary promotion stages:
@@ -48,28 +51,28 @@ Gated canary promotion stages:
4851
* halt advancement if a rolling update is underway
4952
* halt advancement if pods are unhealthy
5053
* increase canary traffic weight percentage from 0% to 5% (step weight)
54+
* call webhooks and check results
5155
* check canary HTTP request success rate and latency
5256
* halt advancement if any metric is under the specified threshold
5357
* increment the failed checks counter
5458
* check if the number of failed checks reached the threshold
5559
* route all traffic to primary
5660
* scale to zero the canary deployment and mark it as failed
57-
* wait for the canary deployment to be updated (revision bump) and start over
61+
* wait for the canary deployment to be updated and start over
5862
* increase canary traffic weight by 5% (step weight) till it reaches 50% (max weight)
5963
* halt advancement while canary request success rate is under the threshold
6064
* halt advancement while canary request duration P99 is over the threshold
6165
* halt advancement if the primary or canary deployment becomes unhealthy
6266
* halt advancement while canary deployment is being scaled up/down by HPA
6367
* promote canary to primary
68+
* copy ConfigMaps and Secrets from canary to primary
6469
* copy canary deployment spec template over primary
6570
* wait for primary rolling update to finish
6671
* halt advancement if pods are unhealthy
6772
* route all traffic to primary
6873
* scale to zero the canary deployment
6974
* mark rollout as finished
70-
* wait for the canary deployment to be updated (revision bump) and start over
71-
72-
You can change the canary analysis _max weight_ and the _step weight_ percentage in the Flagger's custom resource.
75+
* wait for the canary deployment to be updated and start over
7376

7477
For a deployment named _podinfo_, a canary promotion can be defined using Flagger's custom resource:
7578

@@ -248,6 +251,9 @@ kubectl -n test set image deployment/podinfo \
248251
podinfod=quay.io/stefanprodan/podinfo:1.4.0
249252
```
250253

254+
**Note** that Flagger tracks changes in the deployment `PodSpec` but also in `ConfigMaps` and `Secrets`
255+
that are referenced in the pod's volumes and containers environment variables.
256+
251257
Flagger detects that the deployment revision changed and starts a new canary analysis:
252258

253259
```
@@ -336,6 +342,8 @@ Events:
336342
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
337343
```
338344

345+
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
346+
339347
### Monitoring
340348

341349
Flagger comes with a Grafana dashboard made for canary analysis.

‎artifacts/configs/canary.yaml

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
apiVersion: flagger.app/v1alpha3
2+
kind: Canary
3+
metadata:
4+
name: podinfo
5+
namespace: test
6+
spec:
7+
# deployment reference
8+
targetRef:
9+
apiVersion: apps/v1
10+
kind: Deployment
11+
name: podinfo
12+
# the maximum time in seconds for the canary deployment
13+
# to make progress before it is rollback (default 600s)
14+
progressDeadlineSeconds: 60
15+
# HPA reference (optional)
16+
autoscalerRef:
17+
apiVersion: autoscaling/v2beta1
18+
kind: HorizontalPodAutoscaler
19+
name: podinfo
20+
service:
21+
# container port
22+
port: 9898
23+
# Istio gateways (optional)
24+
gateways:
25+
- public-gateway.istio-system.svc.cluster.local
26+
# Istio virtual service host names (optional)
27+
hosts:
28+
- app.iowa.weavedx.com
29+
canaryAnalysis:
30+
# schedule interval (default 60s)
31+
interval: 10s
32+
# max number of failed metric checks before rollback
33+
threshold: 10
34+
# max traffic percentage routed to canary
35+
# percentage (0-100)
36+
maxWeight: 50
37+
# canary increment step
38+
# percentage (0-100)
39+
stepWeight: 5
40+
# Istio Prometheus checks
41+
metrics:
42+
- name: istio_requests_total
43+
# minimum req success rate (non 5xx responses)
44+
# percentage (0-100)
45+
threshold: 99
46+
interval: 1m
47+
- name: istio_request_duration_seconds_bucket
48+
# maximum req duration P99
49+
# milliseconds
50+
threshold: 500
51+
interval: 30s
52+
# external checks (optional)
53+
webhooks:
54+
- name: load-test
55+
url: http://flagger-loadtester.test/
56+
timeout: 5s
57+
metadata:
58+
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

‎artifacts/configs/configs.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: podinfo-config-env
5+
namespace: test
6+
data:
7+
color: blue
8+
---
9+
apiVersion: v1
10+
kind: ConfigMap
11+
metadata:
12+
name: podinfo-config-vol
13+
namespace: test
14+
data:
15+
output: console
16+
textmode: "true"

‎artifacts/configs/deployment.yaml

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: podinfo
5+
namespace: test
6+
labels:
7+
app: podinfo
8+
spec:
9+
minReadySeconds: 5
10+
revisionHistoryLimit: 5
11+
progressDeadlineSeconds: 60
12+
strategy:
13+
rollingUpdate:
14+
maxUnavailable: 0
15+
type: RollingUpdate
16+
selector:
17+
matchLabels:
18+
app: podinfo
19+
template:
20+
metadata:
21+
annotations:
22+
prometheus.io/scrape: "true"
23+
labels:
24+
app: podinfo
25+
spec:
26+
containers:
27+
- name: podinfod
28+
image: quay.io/stefanprodan/podinfo:1.3.0
29+
imagePullPolicy: IfNotPresent
30+
ports:
31+
- containerPort: 9898
32+
name: http
33+
protocol: TCP
34+
command:
35+
- ./podinfo
36+
- --port=9898
37+
- --level=info
38+
- --random-delay=false
39+
- --random-error=false
40+
env:
41+
- name: PODINFO_UI_COLOR
42+
valueFrom:
43+
configMapKeyRef:
44+
name: podinfo-config-env
45+
key: color
46+
- name: SECRET_USER
47+
valueFrom:
48+
secretKeyRef:
49+
name: podinfo-secret-env
50+
key: user
51+
livenessProbe:
52+
exec:
53+
command:
54+
- podcli
55+
- check
56+
- http
57+
- localhost:9898/healthz
58+
initialDelaySeconds: 5
59+
timeoutSeconds: 5
60+
readinessProbe:
61+
exec:
62+
command:
63+
- podcli
64+
- check
65+
- http
66+
- localhost:9898/readyz
67+
initialDelaySeconds: 5
68+
timeoutSeconds: 5
69+
resources:
70+
limits:
71+
cpu: 2000m
72+
memory: 512Mi
73+
requests:
74+
cpu: 100m
75+
memory: 64Mi
76+
volumeMounts:
77+
- name: configs
78+
mountPath: /etc/podinfo/configs
79+
readOnly: true
80+
- name: secrets
81+
mountPath: /etc/podinfo/secrets
82+
readOnly: true
83+
volumes:
84+
- name: configs
85+
configMap:
86+
name: podinfo-config-vol
87+
- name: secrets
88+
secret:
89+
secretName: podinfo-secret-vol

‎artifacts/configs/hpa.yaml

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: autoscaling/v2beta1
2+
kind: HorizontalPodAutoscaler
3+
metadata:
4+
name: podinfo
5+
namespace: test
6+
spec:
7+
scaleTargetRef:
8+
apiVersion: apps/v1
9+
kind: Deployment
10+
name: podinfo
11+
minReplicas: 1
12+
maxReplicas: 4
13+
metrics:
14+
- type: Resource
15+
resource:
16+
name: cpu
17+
# scale up if usage is above
18+
# 99% of the requested CPU (100m)
19+
targetAverageUtilization: 99

‎artifacts/configs/secrets.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: podinfo-secret-env
5+
namespace: test
6+
data:
7+
password: cGFzc3dvcmQ=
8+
user: YWRtaW4=
9+
---
10+
apiVersion: v1
11+
kind: Secret
12+
metadata:
13+
name: podinfo-secret-vol
14+
namespace: test
15+
data:
16+
key: cGFzc3dvcmQ=

‎docs/gitbook/how-it-works.md

+25-21
Original file line numberDiff line numberDiff line change
@@ -97,38 +97,42 @@ the Istio Virtual Service. The container port from the target deployment should
9797

9898
![Flagger Canary Stages](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-steps.png)
9999

100+
A canary deployment is triggered by changes in any of the following objects:
101+
102+
* Deployment PodSpec (container image, command, ports, env, resources, etc)
103+
* ConfigMaps mounted as volumes or mapped to environment variables
104+
* Secrets mounted as volumes or mapped to environment variables
105+
100106
Gated canary promotion stages:
101107

102108
* scan for canary deployments
103-
* creates the primary deployment if needed
104109
* check Istio virtual service routes are mapped to primary and canary ClusterIP services
105110
* check primary and canary deployments status
106-
* halt advancement if a rolling update is underway
107-
* halt advancement if pods are unhealthy
108-
* increase canary traffic weight percentage from 0% to 5% \(step weight\)
111+
* halt advancement if a rolling update is underway
112+
* halt advancement if pods are unhealthy
113+
* increase canary traffic weight percentage from 0% to 5% (step weight)
114+
* call webhooks and check results
109115
* check canary HTTP request success rate and latency
110-
* halt advancement if any metric is under the specified threshold
111-
* increment the failed checks counter
116+
* halt advancement if any metric is under the specified threshold
117+
* increment the failed checks counter
112118
* check if the number of failed checks reached the threshold
113-
* route all traffic to primary
114-
* scale to zero the canary deployment and mark it as failed
115-
* wait for the canary deployment to be updated \(revision bump\) and start over
116-
* increase canary traffic weight by 5% \(step weight\) till it reaches 50% \(max weight\)
117-
* halt advancement if the primary or canary deployment becomes unhealthy
118-
* halt advancement while canary deployment is being scaled up/down by HPA
119-
* halt advancement if any of the webhook calls are failing
120-
* halt advancement while canary request success rate is under the threshold
121-
* halt advancement while canary request duration P99 is over the threshold
119+
* route all traffic to primary
120+
* scale to zero the canary deployment and mark it as failed
121+
* wait for the canary deployment to be updated and start over
122+
* increase canary traffic weight by 5% (step weight) till it reaches 50% (max weight)
123+
* halt advancement while canary request success rate is under the threshold
124+
* halt advancement while canary request duration P99 is over the threshold
125+
* halt advancement if the primary or canary deployment becomes unhealthy
126+
* halt advancement while canary deployment is being scaled up/down by HPA
122127
* promote canary to primary
123-
* copy canary deployment spec template over primary
128+
* copy ConfigMaps and Secrets from canary to primary
129+
* copy canary deployment spec template over primary
124130
* wait for primary rolling update to finish
125-
* halt advancement if pods are unhealthy
131+
* halt advancement if pods are unhealthy
126132
* route all traffic to primary
127133
* scale to zero the canary deployment
128-
* mark the canary deployment as finished
129-
* wait for the canary deployment to be updated \(revision bump\) and start over
130-
131-
You can change the canary analysis _max weight_ and the _step weight_ percentage in the Flagger's custom resource.
134+
* mark rollout as finished
135+
* wait for the canary deployment to be updated and start over
132136

133137
### Canary Analysis
134138

‎docs/gitbook/usage/progressive-delivery.md

+5-2
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
2424
kubectl -n test apply -f ${REPO}/artifacts/loadtester/service.yaml
2525
```
2626

27-
Create a canary custom resource \(replace example.com with your own domain\):
27+
Create a canary custom resource (replace example.com with your own domain):
2828

2929
```yaml
3030
apiVersion: flagger.app/v1alpha3
@@ -146,6 +146,8 @@ Events:
146146
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
147147
```
148148

149+
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
150+
149151
You can monitor all canaries with:
150152

151153
```bash
@@ -181,7 +183,8 @@ Generate latency:
181183
watch curl http://podinfo-canary:9898/delay/1
182184
```
183185

184-
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
186+
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
187+
the canary is scaled to zero and the rollout is marked as failed.
185188

186189
```text
187190
kubectl -n test describe canary/podinfo

‎pkg/apis/flagger/v1alpha3/types.go

+2
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ type CanaryStatus struct {
9393
FailedChecks int `json:"failedChecks"`
9494
CanaryWeight int `json:"canaryWeight"`
9595
// +optional
96+
TrackedConfigs *map[string]string `json:"trackedConfigs,omitempty"`
97+
// +optional
9698
LastAppliedSpec string `json:"lastAppliedSpec,omitempty"`
9799
// +optional
98100
LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`

‎pkg/apis/flagger/v1alpha3/zz_generated.deepcopy.go

+11
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/controller/controller.go

+5
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,11 @@ func NewController(
7474
kubeClient: kubeClient,
7575
istioClient: istioClient,
7676
flaggerClient: flaggerClient,
77+
configTracker: ConfigTracker{
78+
logger: logger,
79+
kubeClient: kubeClient,
80+
flaggerClient: flaggerClient,
81+
},
7782
}
7883

7984
router := CanaryRouter{

0 commit comments

Comments
 (0)