Skip to content

Fix mismatched AirflowConfigException in celery and edge3 - #72437

Merged
dheerajturaga merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-celery-edge3-config-exception
Sep 6, 2026
Merged

dheerajturaga merged 1 commit into
apache:mainfrom
FrankYang0529:airflow-celery-edge3-config-exception

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

Why

  • Update conf imports in providers to use airflow.sdk.configuration #60000 moved providers to from airflow.providers.common.compat.sdk import conf. celery and edge3 took the new conf but left AirflowConfigException coming from airflow.exceptions.
  • The compat conf resolves to the Task SDK parser, which raises airflow.sdk._shared.configuration.exceptions.AirflowConfigException. The handlers name airflow.exceptions.AirflowConfigException, so they never catch what the parser raises.
  • default_celery.py needs both classes in the handler, because get_default_celery_config() also receives core's ExecutorConf on the multi-team path.

How

  • The edge3 plugin and celery_command.py only ever see the compat conf, so AirflowConfigException now comes from the same compat import line.
  • get_default_celery_config() gets the compat conf at module scope and core's ExecutorConf on the multi-team path. Those two raise different classes, so the handler in default_celery.py catches both.

Verification

  • uv run --project providers/celery pytest "providers/celery/tests/unit/celery/executors/test_celery_executor.py" "providers/celery/tests/unit/celery/cli/test_celery_command.py" "providers/edge3/tests/unit/edge3/plugins/test_edge_executor_plugin.py" -v

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:celery provider:edge Edge Executor / Worker (AIP-69) / edge3 labels Sep 2, 2026
@FrankYang0529
FrankYang0529 marked this pull request as ready for review September 3, 2026 12:54
Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 force-pushed the airflow-celery-edge3-config-exception branch from 2275b92 to 682277a Compare September 4, 2026 06:17
@dheerajturaga
dheerajturaga merged commit 80090d2 into apache:main Sep 6, 2026
79 checks passed
@FrankYang0529
FrankYang0529 deleted the airflow-celery-edge3-config-exception branch September 6, 2026 07:38
pierrejeambrun pushed a commit to astronomer/airflow that referenced this pull request Sep 8, 2026
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:celery provider:edge Edge Executor / Worker (AIP-69) / edge3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants