Skip to content

AIP-66: Make DAG callbacks bundle aware - #45860

Merged
jedcunningham merged 7 commits into
apache:mainfrom
astronomer:make-callbacks-bundle-aware
Feb 4, 2025
Merged

jedcunningham merged 7 commits into
apache:mainfrom
astronomer:make-callbacks-bundle-aware

Conversation

@ephraimbuddy

@ephraimbuddy ephraimbuddy commented Jan 21, 2025 •

Copy link
Copy Markdown
Contributor

This involves using relative paths in the callbacks, resolving the full path and using it to queue the callback in the file processor process.

Closes: #45496

Comment thread airflow/dag_processing/manager.py Outdated
Comment thread airflow/dag_processing/manager.py Outdated
Comment thread airflow/dag_processing/processor.py
Comment thread airflow/models/dag.py Outdated
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch from 3ab1310 to 1bd1551 Compare January 23, 2025 00:46
Comment thread airflow/dag_processing/manager.py Outdated
Comment thread airflow/dag_processing/manager.py Outdated
Comment thread airflow/dag_processing/manager.py Outdated
Comment thread airflow/dag_processing/manager.py Outdated
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch 5 times, most recently from 9ffb322 to 06c46f7 Compare January 28, 2025 11:50
@ephraimbuddy
ephraimbuddy marked this pull request as ready for review January 28, 2025 11:53
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch from 879774e to 0f6beb2 Compare January 28, 2025 12:49
Comment thread airflow/triggers/base.py
Comment thread airflow/callbacks/callback_requests.py Outdated
Comment thread tests/dag_processing/test_manager.py Outdated
Comment thread tests/dag_processing/test_manager.py Outdated
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch from 0f6beb2 to b48dbaf Compare January 29, 2025 09:03
Comment thread airflow/callbacks/callback_requests.py Outdated
Comment thread airflow/triggers/base.py Outdated
Comment thread airflow/triggers/base.py
Comment thread airflow/jobs/scheduler_job_runner.py Outdated
Comment thread airflow/jobs/scheduler_job_runner.py Outdated
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch from b48dbaf to 8c8a21f Compare January 30, 2025 18:15
Comment thread airflow/jobs/scheduler_job_runner.py Outdated
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch 4 times, most recently from ec5f309 to b3438c1 Compare February 3, 2025 20:00
This involves using relative paths in the callbacks, resolving the full path and using the
it to queue the callback in the file processor process.

Don't serialize dag if running callback

Refactor codes and fix tests

update callback test

Fix typing

Don't use relative fileloc yet

Add test

apply suggestions from code review

Fix some tests and require bundle_version in DagCallbackRequest

Resolve conflicts and fix tests

join load dagversion as part of TI in process executor event
@ephraimbuddy
ephraimbuddy force-pushed the make-callbacks-bundle-aware branch from b3438c1 to 50be7e9 Compare February 3, 2025 21:13
@jedcunningham jedcunningham added full tests needed We need to run full set of tests for this PR to merge area:dag-bundles AIP-66 labels Feb 4, 2025
@jedcunningham jedcunningham reopened this Feb 4, 2025
@jedcunningham
jedcunningham merged commit 8c3ce93 into apache:main Feb 4, 2025
@jedcunningham
jedcunningham deleted the make-callbacks-bundle-aware branch February 4, 2025 01:44
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
This involves using relative paths in the callbacks, resolving the full path and using the
it to queue the callback in the file processor process.
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
This involves using relative paths in the callbacks, resolving the full path and using the
it to queue the callback in the file processor process.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dag-bundles AIP-66 area:scheduler full tests needed We need to run full set of tests for this PR to merge

Projects

Development

Successfully merging this pull request may close these issues.

AIP-66: Make callbacks bundle aware

2 participants