Skip to content

Add validate trigger events in dbt cloud deferrable tasks - #69739

Merged
josh-fell merged 4 commits into
apache:mainfrom
takayoshi-makabe:add-validate-trigger-events-in-dbt-cloud-deferrable-tasks
Sep 24, 2026
Merged

josh-fell merged 4 commits into
apache:mainfrom
takayoshi-makabe:add-validate-trigger-events-in-dbt-cloud-deferrable-tasks

Conversation

@takayoshi-makabe

Copy link
Copy Markdown
Contributor

DbtCloudRunJobOperator.execute_complete and DbtCloudJobRunSensor.execute_complete used the trigger event without validating it:

  • event=None: happens when a deferred task resumes without a payload — a custom or subclassed trigger that yields TriggerEvent(None), or an event payload lost between the triggerer and the worker. Both methods crashed with an opaque TypeError: 'NoneType' object is not subscriptable.
  • Unrecognised status: happens on triggerer/worker version skew (a task defers under one provider version and resumes under another whose trigger emits a different status vocabulary) or with a custom trigger. DbtCloudRunJobTrigger emits success/cancelled/error/timeout, but both execute_complete methods branch deny-list style (only specific failure statuses raise, everything else succeeds), so an unknown status fell through into the success path — the task was marked SUCCESS even though the dbt Cloud job outcome was unknown.
  • The sensor additionally never checked timeout at all — it only raised on error/cancelled, so a job run that timed out while deferred was silently marked SUCCESS.

This adds a validate_execute_complete_event() helper (with a dedicated DbtCloudTriggerEventException), applied at the top of both execute_complete methods. Same fix shape as the anthropic (#69379) and OpenAI (#69506) providers: the None check mirrors the amazon provider's helper of the same name, and the status check plays the role amazon's call sites cover with their if validated_event["status"] != "success": raise allow-list branching.

The sensor's fix also replaces a bare raise AirflowException() (no message) with DbtCloudJobRunException(event["message"]), which both fixes the missing timeout handling and surfaces the trigger's actual message instead of an empty exception.

The operator keeps its existing per-status branching (cancelled/error/timeout each have distinct messages, and timeout additionally attempts job cancellation) since validate_execute_complete_event() already closes the status set to the four values above.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude code


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 11, 2026

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the contribution :)

Comment thread providers/dbt/cloud/src/airflow/providers/dbt/cloud/hooks/dbt.py

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM.

nit: Title typo: tirgger -> trigger

@takayoshi-makabe takayoshi-makabe changed the title Add validate tirgger events in dbt cloud deferrable tasks Add validate trigger events in dbt cloud deferrable tasks Sep 12, 2026
@takayoshi-makabe

Copy link
Copy Markdown
Contributor Author

@aaron-y-chen
Sorry, I fixed title.

@aaron-y-chen

Copy link
Copy Markdown
Contributor

No need to apologize. Thanks for your hard work! 😄

Comment thread providers/dbt/cloud/src/airflow/providers/dbt/cloud/hooks/dbt.py
@josh-fell
josh-fell merged commit 1ac02ad into apache:main Sep 24, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants