-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explore and add static checks for DAGs for early detection of common issues #43176
Comments
In the context of this issue, DAG linting as mentioned by this article (shared by @potiuk on slack) can also be explored: Inspired by above article, may be we can have commands like |
I feel it's a bit different. 🤔 But for |
Good to see new airflow-specific Ruff rules being added (AIR3##). Nice work @Lee-W! |
This is definitely something I'd find immensely useful. In addition to the article linked in the earlier comments, there are some decent prior attempts at doing something similar with static DAG code checks (see links below); however, I think the major benefit for making this feature native to Airflow is that it can rely directly on the built-in DAG parsing logic. Some of the examples I was referring to:
|
We could probably start discussing what the best practice is after Airflow 3.0 is released (best practice now might not reflect the latest best practice we want then) |
Yes that makes sense, let's add support for best practices linting for Airflow 3.x onwards. I just took a quick opinion and got a good concensus for 3.x onwards on slack. |
@Lee-W would you be interested in picking up this issue? Because I suppose you have good context on Airflow static checks by adding Ruff rules for it, among other things. |
Yep, implementing them is interesting, and I would definitely be interested in adding more rules haha |
I'll probably focus most on implementing rules, but will need everyone's suggestion on what the best practices we want 🙂 |
Perhaps we should start a shared doc to raise ideas? Personally, I'd love some checks related to XCOMs, and in particular in the context of task groups, to ensure they're unpacked/deserialized properly. Also, if possible, a check that custom classes returned by dags are included in the |
we could probably just discuss here and after we have some rough idea, we can organize them to something like #41641 |
Idea: |
Idea: # Before:
some_task = SomeOperator(task_id="some_task", ...)
some_other_task = SomeOtherOperator(
templated_field="{{ ti.xcom_pull(task_ids='some_task') }}",
)
# After:
some_task = SomeOperator(...) # Note the task id no longer has to be specified explicitly.
some_other_task = SomeOtherOperator(
templated_field=some_task.output,
) ...and equivalently for tasks. |
Some ideas from this blog post:
|
Some additional best practices rule suggestions:
|
One other lint check idea came to mind:
|
I'm not sure Ruff supports multi-file checks, so I suspect this might only be applicable to situations where multiple DAGs are defined in a single script. IIRC this can be detected by a DagBag unit test.
I think this too belongs in the realm of checks to be added to
Could you please elaborate (mainly out of curiosity):
All in all I these are some good ideas to be included in |
Thanks @Dev-iL for referencing my blog post 🙏 Another thing we can add is consider implementing a rule, adding to the third rule above, that dag id should be all lower case characters. Since DAG IDs should match the file names, enforcing this one is making sure DAG files to be compatible with the Python convention for file names. |
|
@Dev-iL Thank you for pointing me towards the DagBag unit testing page, I wasn't aware that was an available approach. You're correct that the DagBag unit testing would cover unique DAG ID enforcement. As for checking against user-defined cluster policies, I think the DagBag unit testing could cover that, but the cluster policy docs are not clear if that is evaluated during the DagBag loading, or during the actual DAG/task scheduling. TL;DR: I think the most broadly useful static check would be the checks for problematic top-level code (e.g. network calls,
|
Description
As per users' feedback in the Airflow Debugging Survey 2024, 48.3% of respondents chose early issue detection during execution as one of their top 2 choices.
Use case/motivation
Goal of this issue:
Related issues
Parent Issue: #40975
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: