Build DataprocCreateClusterOperator cluster_config from legacy kwargs after rendering by bujjibabukatta · Pull Request #70922 · apache/airflow

bujjibabukatta · 2026-08-01T17:17:18Z

Summary
DataprocCreateClusterOperator's deprecated keyword-based cluster
construction built cluster_config in init, using individually
templated kwargs (zone, cluster_name, project_id, ...) before rendering.

Root cause
ClusterGenerator(**kwargs).make() ran in init. These kwargs aren't
declared as template fields, so Jinja expressions passed this way never
got rendered before being baked into cluster_config.

Fix
Stash the raw kwargs in a new template field (_legacy_cluster_kwargs)
instead of building cluster_config immediately. Being a template field, it
survives DAG serialization and gets rendered automatically. cluster_config
is now built in execute(), after rendering.

related: #70296

Was generative AI tooling used ?

Yes - Claude

Generated-by: Claude following the guidelines

SameerMesiah97

Left 2 comments.

bujjibabukatta · 2026-08-04T12:35:54Z

Hi @SameerMesiah97 thanks for catching both — fixed. Wrapped the operator
construction in pytest.warns(AirflowProviderDeprecationWarning) in both
tests, and reformatted to one argument per line.

Separately: Static checks keeps failing on validate-operators-init, flagging
files unrelated to this PR (e.g. glacier.py, whose init is a plain
assignment with no logic at all). I've reset onto current main and
re-cherry-picked a few times now — it passes briefly then fails the same way
again, which looks like a merge-base/timing issue against how fast main is
moving, not anything in this diff. Would appreciate a quick look or merge to
break the cycle.

… after rendering

…mplate field

…a/airflow into fix/apache#70296-1

…che#70296-1

… after rendering

bujjibabukatta · 2026-08-21T15:46:03Z

Hi @olegkachur-e, @SameerMesiah97 can you please review and approve pull request ?

shahar1 · 2026-09-12T19:50:11Z

            if project_id is None:
                raise AirflowException(
                    "project_id argument is required when building cluster from keywords parameters"
                )
            kwargs["project_id"] = project_id
-            cluster_config = ClusterGenerator(**kwargs).make()


Why changing that?
kwargs is not in template_fields

shahar1 · 2026-09-12T19:51:48Z

@@ -726,19 +725,22 @@ def __init__(
            if "params" in kwargs:
                del kwargs["params"]

-            # Create cluster object from kwargs
            if project_id is None:
                raise AirflowException(
                    "project_id argument is required when building cluster from keywords parameters"
                )
            kwargs["project_id"] = project_id


project_id is a template field, so this statement should be dropped and we should rely only on self.project_id

shahar1 · 2026-09-12T19:53:07Z

+    def _build_cluster_config_from_legacy_kwargs(self) -> dict:
+        """Build cluster_config from legacy keyword args, called post-render in execute()."""
+        if self._legacy_cluster_kwargs is None:
+            raise RuntimeError("The _legacy_cluster_kwargs should be set here!")


The message should say what is wrong ("here" is not clear enough)

shahar1 · 2026-09-12T20:00:03Z

+
+    @mock.patch(DATAPROC_PATH.format("Cluster.to_dict"))
+    @mock.patch(DATAPROC_PATH.format("DataprocHook"))
+    def test_deprecated_kwargs_cluster_config_built_in_execute(self, mock_hook, to_dict_mock):


I would add some tests for the template fields (examples are available in GCP's test_compute.py and test_cloud_storage_transfer_service.py)

…_id injection, remove exemption

…g template

VladaZakharova · 2026-09-23T10:28:25Z

@@ -685,6 +685,7 @@ class DataprocCreateClusterOperator(GoogleCloudBaseOperator):
        "labels",
        "gcp_conn_id",
        "impersonation_chain",
+        "_legacy_cluster_kwargs",


Looks like previously legacy kwargs built cluster_config during init. Now cluster_config remains None until execute().
Normal task execution should remain compatible, but code that reads operator.cluster_config before execution—cluster policies, subclasses, custom validation, or tests—will observe different behavior. Invalid legacy configuration also moves from Dag parsing time to task execution time.

Also DataprocCreateClusterOperator remains in validate_operators_init_exemptions.txt. but this PR does not finish this operator’s part of issue #70296.
The new _legacy_cluster_kwargs field is synthesized in init, so it does not follow the validator’s normal direct-assignment rule. The implementation and exemption strategy need to be resolved together.

VladaZakharova · 2026-09-23T10:29:11Z

-            cluster_config = ClusterGenerator(**kwargs).make()
+
+            # Defer building cluster_config until execute(), after templated fields render.
+            self._legacy_cluster_kwargs: dict | None = dict(kwargs)


this captures more than Dataproc legacy parameters, including params, dag, task_group, callbacks, executor configuration, and other BaseOperator values.
Only ClusterGenerator arguments should be captured, no?

bujjibabukatta requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, jscheffl, potiuk and shahar1 as code owners August 1, 2026 17:17

boring-cyborg Bot added area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:google Google (including GCP) related issues labels Aug 1, 2026

SameerMesiah97 reviewed Aug 1, 2026

View reviewed changes

Comment thread providers/google/tests/unit/google/cloud/operators/test_dataproc.py Outdated

Comment thread providers/google/tests/unit/google/cloud/operators/test_dataproc.py Outdated

bujjibabukatta force-pushed the fix/#70296-1 branch 3 times, most recently from 82398f0 to e12ee19 Compare August 4, 2026 11:15

olegkachur-e reviewed Aug 4, 2026

View reviewed changes

Comment thread providers/google/src/airflow/providers/google/cloud/operators/dataproc.py Outdated

bujjibabukatta added 2 commits August 6, 2026 21:58

Build DataprocCreateClusterOperator cluster_config from legacy kwargs…

02dbb41

… after rendering

Wrap deprecated-kwargs construction in pytest.warns per review

6dbb39b

bujjibabukatta force-pushed the fix/#70296-1 branch from 9d1b55f to 6dbb39b Compare August 6, 2026 16:30

bujjibabukatta added 5 commits August 6, 2026 22:22

Trigger CI

9200e61

Merge branch 'main' into fix/apache#70296-1

a9f7827

Keep DataprocCreateClusterOperator exemption; revert legacy-kwargs te…

ea18f30

…mplate field

Merge branch 'fix/apache#70296-1' of https://github.com/bujjibabukatt…

d6fef4d

…a/airflow into fix/apache#70296-1

Merge branch 'main' of https://github.com/apache/airflow into fix/apa…

c2ee766

…che#70296-1

bujjibabukatta force-pushed the fix/#70296-1 branch 2 times, most recently from df95d98 to 4561488 Compare August 20, 2026 10:23

Build DataprocCreateClusterOperator cluster_config from legacy kwargs…

99b08c4

… after rendering

bujjibabukatta force-pushed the fix/#70296-1 branch from 4561488 to 99b08c4 Compare August 21, 2026 04:25

shahar1 requested changes Sep 12, 2026

View reviewed changes

Address review: render legacy kwargs via template field, drop project…

d63b680

…_id injection, remove exemption

bujjibabukatta force-pushed the fix/#70296-1 branch from f143916 to d63b680 Compare September 17, 2026 08:48

Fix DataprocCreateClusterOperator: don't strip params before renderin…

940234b

…g template

VladaZakharova suggested changes Sep 23, 2026

View reviewed changes

shahar1 mentioned this pull request Sep 24, 2026

Move template-field validation/transformation out of operator __init__ (exemption-list burn-down) #70296

Closed

78 tasks

Navigation Menu

Build DataprocCreateClusterOperator cluster_config from legacy kwargs after rendering - #70922

Build DataprocCreateClusterOperator cluster_config from legacy kwargs after rendering#70922
bujjibabukatta wants to merge 10 commits into
apache:mainfrom
bujjibabukatta:fix/#70296-1

bujjibabukatta commented Aug 1, 2026 •

edited by shahar1

Loading

Uh oh!

SameerMesiah97 left a comment

Uh oh!

Uh oh!

Uh oh!

bujjibabukatta commented Aug 4, 2026

Uh oh!

Uh oh!

bujjibabukatta commented Aug 21, 2026 •

edited

Loading

Uh oh!

shahar1 Sep 12, 2026

Uh oh!

shahar1 Sep 12, 2026

Uh oh!

shahar1 Sep 12, 2026

Uh oh!

shahar1 Sep 12, 2026

Uh oh!

VladaZakharova Sep 23, 2026

Uh oh!

VladaZakharova Sep 23, 2026

Uh oh!

VladaZakharova Sep 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

bujjibabukatta commented Aug 1, 2026 • edited by shahar1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SameerMesiah97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bujjibabukatta commented Aug 4, 2026

Uh oh!

Uh oh!

bujjibabukatta commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shahar1 Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

shahar1 Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

shahar1 Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

shahar1 Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

VladaZakharova Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

VladaZakharova Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

VladaZakharova Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bujjibabukatta commented Aug 1, 2026 •

edited by shahar1

Loading

bujjibabukatta commented Aug 21, 2026 •

edited

Loading