Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-83 Amendment refactor reprocess behaviour in backfill #46278

Open
2 tasks done
vatsrahul1001 opened this issue Jan 30, 2025 · 0 comments
Open
2 tasks done

AIP-83 Amendment refactor reprocess behaviour in backfill #46278

vatsrahul1001 opened this issue Jan 30, 2025 · 0 comments
Assignees
@vatsrahul1001
Labels
area:backfill Specifically for backfill related kind:feature Feature Requests
Milestone
Airflow 3.0.0

Comments

@vatsrahul1001
Copy link
Collaborator

Description

We will encounter an issue when attempting to create a backfill with the reprocess behavior set to completed or failed, as it will trigger a unique constraint error on logical_date due to the restored uniqueness constraint.

To align with the old (2.x) behavior for existing runs that need to be reprocessed, we should ensure that when reprocessing results in a new run, the existing run is cleared. Instead of deleting the row, we update the necessary fields—similar to what was done in 2.x. This at least includes changing the state, but likely also updating start_date and possibly other relevant fields.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@vatsrahul1001 vatsrahul1001 added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Jan 30, 2025
@vatsrahul1001 vatsrahul1001 self-assigned this Jan 30, 2025
@phanikumv phanikumv removed the needs-triage label for new issues that we didn't triage yet label Jan 30, 2025
@phanikumv phanikumv added this to the Airflow 3.0.0 milestone Jan 30, 2025
@phanikumv phanikumv moved this to Todo in AIP-83 amendment Jan 30, 2025
@dosubot dosubot bot added the area:backfill Specifically for backfill related label Jan 30, 2025
@vatsrahul1001 vatsrahul1001 mentioned this issue Jan 30, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vatsrahul1001 vatsrahul1001

Labels
area:backfill Specifically for backfill related kind:feature Feature Requests
Projects
AIP-83 amendment
Status: Todo
Milestone
Airflow 3.0.0
Development

No branches or pull requests
2 participants
@vatsrahul1001 @phanikumv