[FLINK-37515] Basic support for Blue/Green deployments #969

schongloo · 2025-04-08T16:29:57Z

What is the purpose of the change

This pull request adds basic support for Blue/Green deployments as outlined by FLIP-503 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677648).

Brief change log

A new FlinkBlueGreenDeploymentController controller, with the capability of managing the lifecycle of these deployments.
A new corresponding CRD is introduced
The new FlinkBlueGreenDeploymentController will manage this CRD and hide from the user the details of the actual Blue/Green (Active/StandBy) jobs.
Delegate the lifecycle of the actual Jobs to the existing FlinkDeployment controller.

Verifying this change

Test to verify a single basic deployment
Test to verify a proper transition Blue -> Green
Test to verify correct behavior when encountering an error before a transition
Test to verify correct behavior when encountering an error during a transition

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changes to the CustomResourceDescriptors: no
Core observer or reconciler logic that is regularly executed: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs / JavaDocs. UML diagrams for all scenarios added under docs/content/docs

…ption and never reach FINISHED in the CR

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

davidradl · 2025-04-10T10:59:56Z

I am concerned about the number of TODOs. Things like dealing with errors and timing conditions should be in place in the initial drop of code - unless we can mark this capability as beta or the like. WDYT @gyfora ?

…d-transition.

schongloo · 2025-04-11T17:45:41Z

@davidradl thanks for your review/comments. I've cleaned up all the outdated/invalid TODOs which pointed me 1 missing unit test, added. At this point the few remaining TODOs do not interfere with the main functionality and are not critical/open for discussion.

… BASIC is supported).Tests to confirm the deployments are deleted accurately after the specified deletion delay.

…xceptions.

hansh0801 · 2025-04-23T09:05:36Z

👍

hansh0801 · 2025-04-29T05:44:12Z

when this feature gonna be released?

schongloo · 2025-04-30T13:25:00Z

Hi @hansh0801, trying to release it ASAP but I also want to make sure the logic and contracts are as robust as possible to minimize changes later. Looks pretty stable in my current testing and I've reopened it for review. Thanks!

hansh0801 · 2025-04-30T13:30:33Z

cool ! and I'm also waiting for blue green phase 2 :)
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677650

schongloo · 2025-04-30T18:20:23Z

@hansh0801 indeed... Phase 2 will take a little bit longer due to its complexity but already working on it.

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/TransitionMode.java

...src/main/java/org/apache/flink/kubernetes/operator/api/spec/FlinkDeploymentTemplateSpec.java

...ain/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentStatus.java

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

gyfora · 2025-05-07T12:58:37Z

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

+        if (deploymentStatus == null) {
+            deploymentStatus = new FlinkBlueGreenDeploymentStatus();
+            return patchStatusUpdateControl(
+                            bgDeployment,
+                            deploymentStatus,
+                            FlinkBlueGreenDeploymentState.INITIALIZING_BLUE,
+                            null)
+                    .rescheduleAfter(100);


The BlueGreenDeployment could simply implement initStatus and this can be removed completely.

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

gyfora · 2025-05-07T13:08:16Z

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

+    private boolean hasSpecChanged(
+            FlinkBlueGreenDeploymentSpec newSpec, FlinkBlueGreenDeploymentStatus deploymentStatus) {
+
+        String lastReconciledSpec = deploymentStatus.getLastReconciledSpec();
+        String newSpecSerialized = SpecUtils.serializeObject(newSpec, "spec");
+
+        // TODO: in FLIP-504 check here the TransitionMode has not been changed
+
+        return !lastReconciledSpec.equals(newSpecSerialized);


We may want to use the DiffBuilder/ existing utilities to get the type of the spec change here and make sure we don't trigger deployments on non-upgrade changes.

Things like operator config changes etc should not go through the Blue/Green flow I believe but simply applied on the currently active deployment.

Sure, let me get familiar with the DiffBuilder and see how to incorporate it

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

davidradl · 2025-05-07T14:32:09Z

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java

+    TRANSITIONING_TO_BLUE,
+
+    /** Identifies the system is transitioning from "Blue" to "Green". */
+    TRANSITIONING_TO_GREEN,


what state are we in during shutdown?

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java

davidradl · 2025-05-07T14:45:22Z

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java

+            return objectMapper.writeValueAsString(wrapper);
+        } catch (JsonProcessingException e) {
+            throw new RuntimeException(
+                    "Could not serialize " + wrapperKey + ", this indicates a bug...", e);


nit: should be include object.toString() in the error also?

Yes it could be done, I just kept the same existing pattern as the rest of the functions in this class.

helm/flink-kubernetes-operator/templates/rbac.yaml

…fied

maheepm-lyft · 2025-06-16T20:08:06Z

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

gyfora · 2025-06-19T07:45:06Z

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

@maheepm-lyft , @schongloo is currently on holiday but once he is back we can try to merge this as soon as reasonably possible :)

In the meantime you can help us by trying out the current version and by providing a code review for the PR, that can definitely speed things up.

gyfora · 2025-06-23T07:40:59Z

@schongloo can you please re-open this PR against the main branch?

maheepm-lyft · 2025-06-30T22:03:46Z

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

@maheepm-lyft , @schongloo is currently on holiday but once he is back we can try to merge this as soon as reasonably possible :)

In the meantime you can help us by trying out the current version and by providing a code review for the PR, that can definitely speed things up.

Great thanks! Taking us a bit of time to test this out in our stack, but we're working on it and I'll post any comments here.

schongloo · 2025-07-01T18:38:24Z

@schongloo can you please re-open this PR against the main branch?

Done @gyfora !

gyfora and others added 4 commits February 21, 2025 16:02

[release] Update docs config for release branch

be8e899

[FLINK-37372] Fix infinite loop bug in savepoint error handling

415ace4

[FLINK-37370] [Observer] Finished batch jobs throw ReconciliationExce…

01d2440

…ption and never reach FINISHED in the CR

FLIP-503 first batch

ba62f07

schongloo mentioned this pull request Apr 8, 2025

[FLINK-37515] Basic support for Blue/Green deployments #954

Closed

davidradl reviewed Apr 10, 2025

View reviewed changes

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java Outdated Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java Outdated Show resolved Hide resolved

schongloo added 2 commits April 11, 2025 10:38

Addressing PR comments. Adding unit test to cover for Spec changes mi…

eb5b24f

…d-transition.

Addressing PR comments/cleaning up comments.

75d787e

schongloo requested a review from davidradl April 11, 2025 17:41

schongloo added 2 commits April 14, 2025 00:28

Adding a TransitionMode to the CRD/API for the future (currently only…

8addc70

… BASIC is supported).Tests to confirm the deployments are deleted accurately after the specified deletion delay.

Addressing the test sync issues that caused intermittent NullPointerE…

6ecc28c

…xceptions.

schongloo marked this pull request as draft April 15, 2025 18:06

Simplifying the retry/abort logic.

1df7747

Addressing edge case restarting after the first deployment fails.

3690b0d

schongloo marked this pull request as ready for review April 30, 2025 13:22

schongloo changed the base branch from release-1.11 to main April 30, 2025 21:14

schongloo changed the base branch from main to release-1.11 April 30, 2025 22:33

schongloo added 3 commits May 5, 2025 09:41

Undoing accidental test removal

24f65e8

Defensive code vs. non-BASIC TransitionMode (only for FLIP-503)

18da43c

Updated rbac.yaml

7d5072a

gyfora requested changes May 7, 2025

View reviewed changes

davidradl reviewed May 7, 2025

View reviewed changes

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java Outdated Show resolved Hide resolved

davidradl reviewed May 7, 2025

View reviewed changes

helm/flink-kubernetes-operator/templates/rbac.yaml Outdated Show resolved Hide resolved

schongloo added 10 commits May 13, 2025 11:36

Addressing PR comments

27ae567

Fixing accidental comment

7bcb244

More PR comments addressed

4834517

Clean up

4f4c39f

Updated Blue/Green CRD

f9245aa

Using the initial savepoint path, on first deployments only, if speci…

dcce6c8

…fied

Clean up

d4b31ea

Clean up

f04723f

Clean up

3698bb3

Addressing PR comments

d62260b

schongloo added 3 commits July 1, 2025 11:05

Merging main (release 1.13)

f203cd0

Remove scale subresource from blue green deployments

ba6197a

Remove scale subresource from blue green deployments

3bbd618

schongloo changed the base branch from release-1.11 to main July 1, 2025 18:36

schongloo requested review from gyfora and davidradl July 1, 2025 18:37

Merge branch 'main' into release-1.11-bluegreen-flip503

0eac03b

[FLINK-37515] Basic support for Blue/Green deployments #969

Are you sure you want to change the base?

[FLINK-37515] Basic support for Blue/Green deployments #969

Conversation

schongloo commented Apr 8, 2025

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidradl commented Apr 10, 2025

Uh oh!

schongloo commented Apr 11, 2025

Uh oh!

hansh0801 commented Apr 23, 2025

Uh oh!

hansh0801 commented Apr 29, 2025

Uh oh!

schongloo commented Apr 30, 2025

Uh oh!

hansh0801 commented Apr 30, 2025

Uh oh!

schongloo commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gyfora May 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gyfora May 7, 2025

Choose a reason for hiding this comment

Uh oh!

schongloo May 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidradl May 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidradl May 7, 2025

Choose a reason for hiding this comment

Uh oh!

schongloo Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maheepm-lyft commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gyfora commented Jun 19, 2025

Uh oh!

gyfora commented Jun 23, 2025

Uh oh!

maheepm-lyft commented Jun 30, 2025

Uh oh!

schongloo commented Jul 1, 2025

Uh oh!

Uh oh!

maheepm-lyft commented Jun 16, 2025 •

edited

Loading