Skip to content

Make SparkSubmitOperator durable execution inert below Airflow 3.3 - #71534

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:spark-durable-execution-compat-fixes-main
Aug 18, 2026
Merged

amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:spark-durable-execution-compat-fixes-main

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

durable was already wired straight into the operator's YARN cluster-mode validation and, before this, was silently accepted below 3.3 with no signal that it does nothing there since this provider still supports Airflow 2.11 through 3.x and ResumableJobMixin (and the task state store it relies on) only exists on 3.3+.

The compat stub now warns if durable is set explicitly below 3.3 and always resolves it to False, matching the same fix already applied to #71531

reconnect_on_retry -- the parameter's original name, introduced in the same PR as ResumableJobMixin itself and renamed to durable almost immediately -- gets a plain map-and-deprecate rather than the legacy-flag treatment Glue and KPO need. It was never a distinct pre-existing mechanism with its own below-3.3 behavior to preserve, so telling a caller on any version to "use durable instead" is not bad advice the way it would be for a flag with real legacy behavior. The
deprecation warning fires on every Airflow version regardless, since removal happens on this provider's own release schedule, not the caller's.

The YARN yarn_track_via_rm_api requirement needed no code change: it already reads self.durable, so once durable is properly inert below 3.3, the requirement naturally stops applying there too -- there's no durable-execution prerequisite to enforce when durable execution itself isn't active.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@amoghrajesh

Copy link
Copy Markdown
Contributor Author

cc @nailo2c would love to get a review if you can

@amoghrajesh
amoghrajesh merged commit f8b8461 into apache:main Aug 18, 2026
79 checks passed
@amoghrajesh
amoghrajesh deleted the spark-durable-execution-compat-fixes-main branch August 18, 2026 05:20
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants