Created new exception, that provides details of each step of a job ru… #48466
+103
−12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n in the logs when a job fails (apache#46923)
2 tasks
…ncelled job status
jason810496
reviewed
Apr 2, 2025
Comment on lines
-216
to
+221
| raise DbtCloudJobRunException(f"Job run {self.run_id} has failed or has been cancelled.") | ||
| # Gather job run details (including run steps), so we can output the logs from each step | ||
| run_details = self.hook.get_job_run( | ||
| run_id=self.run_id, account_id=self.account_id, include_related=["run_steps"] | ||
| ).json()["data"] | ||
|
|
||
| raise DbtCloudJobRunDetailsException(self.run_id, run_details) |
There was a problem hiding this comment.
We can modularize this part as _raise_dbt_job_run_detail_exception method.
Comment on lines
-242
to
+253
| raise DbtCloudJobRunException(f"Job run {self.run_id} has failed or has been cancelled.") | ||
| # Gather job run details (including run steps), so we can output the logs from each step | ||
| run_details = self.hook.get_job_run( | ||
| run_id=self.run_id, account_id=self.account_id, include_related=["run_steps"] | ||
| ).json()["data"] | ||
|
|
||
| raise DbtCloudJobRunDetailsException(self.run_id, run_details) | ||
|
|
There was a problem hiding this comment.
Then the method can be reused here and several parts below.
Comment on lines
+150
to
+175
| # For extracting only a subset of the higher level job details | ||
| # id is the run_id | ||
| run_data_keys = [ | ||
| "id", | ||
| "job_id", | ||
| "is_error", | ||
| "dbt_version", | ||
| "finished_at", | ||
| "finished_at_humanized", | ||
| "run_duration", | ||
| "run_duration_humanized", | ||
| ] | ||
| # For extracting only a subset of the run step details | ||
| # id is the step id | ||
| run_steps_keys = [ | ||
| "run_id", | ||
| "id", | ||
| "index", | ||
| "name", | ||
| "status", | ||
| "status_humanized", | ||
| "duration", | ||
| "duration_humanized", | ||
| "logs", | ||
| "debug_logs", | ||
| ] |
There was a problem hiding this comment.
Nit: How about make these list for filtering keys as properties of the exception class?
|
@jason810496 Thank you for the review. I'm AFK until next Tuesday. I will address your comments then :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #46923
Created new exception
DbtCloudJobRunDetailsException, which surfaces the job run logs from dbt Cloud, whenever a job run fails or is cancelled.Side-Note: I did notice that when the operator is
deferred, it finishes pre-maturely (i.e. the trigger fires an event before the job is in a terminal status). This wasn't an issue with the operator code in branchv2-10-stable, but is a current issue inmain. I can try to sneak the change in here, or open a new issue to resolve.Current Task Log Output
... File "/opt/airflow/airflow/providers/dbt/cloud/operators/dbt.py", line 182, in execute raise DbtCloudJobRunException(f"Job run {self.run_id} has failed or has been cancelled.") airflow.providers.dbt.cloud.hooks.dbt.DbtCloudJobRunException: Job run 70471831332476 has failed or has been cancelled.New Task Log Output
EDIT: Used chatgpt to format the above traceback: