Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-83: Restore Uniqueness Constraint on Logical Date, Make It Nullable #46295

Merged
merged 47 commits into from
Feb 6, 2025

Conversation

vatsrahul1001
Copy link
Collaborator

@vatsrahul1001 vatsrahul1001 commented Jan 30, 2025

closes:#46187


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@vatsrahul1001
Copy link
Collaborator Author

Need to look at failures.

@uranusjr
Copy link
Member

uranusjr commented Feb 5, 2025

I added a couple of changes to fix tests in ways I think make more sense. This looks good to me except the parts involving backfill; deferring those to Daniel.

Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the above comments I think we should add more test for logical_dates as null especially in models.

tests/models/test_dagrun.py Show resolved Hide resolved
@dstandish
Copy link
Contributor

Let's take the backfill part of this out; we can just do minimal changes to backfill such that tests are passing. Backfill can be broken if necessary as a result of this and fixed by sneha's PR. But we need to get the constraint change in there.

@uranusjr
Copy link
Member

uranusjr commented Feb 5, 2025

Let’s take out the changes in backfill.py from this PR so we can merge the model changes first. Everything else sort of depends on them. This means backfilling would break if you hit a duplicate logical date, but I think it’s fine as long as we fix it before the final release.

I’ll do that tomorrow if nobody beats me to it.

@vatsrahul1001 vatsrahul1001 changed the title AIP-83: Restore Uniqueness Constraint on Logical Date, Make It Nullable & Refactor Backfill Reprocessing AIP-83: Restore Uniqueness Constraint on Logical Date, Make It Nullable Feb 5, 2025
@vatsrahul1001
Copy link
Collaborator Author

@uranusjr @dstandish I have removed backfill related changes from this PR. We should be good to merge this after CI is green

@vatsrahul1001 vatsrahul1001 requested review from dstandish, sunank200 and Lee-W and removed request for dstandish February 5, 2025 16:55
@dstandish
Copy link
Contributor

@uranusjr @dstandish I have removed backfill related changes from this PR. We should be good to merge this after CI is green

ok cool

Copy link
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @uranusjr if it also looks good to me, I think we could probably just merge it?

Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made a change in tests, I’ll merge this once everything is green.

@uranusjr uranusjr merged commit 9aa3309 into apache:main Feb 6, 2025
45 checks passed
@uranusjr uranusjr deleted the refactor-backfill-reprocess-logic branch February 6, 2025 08:32
@ashb
Copy link
Member

ashb commented Feb 6, 2025

providers/tests/amazon/aws/operators/test_dms.py:503: in test_template_fields_native
    context = ti.get_template_context(session)
airflow/models/taskinstance.py:3242: in get_template_context
    return _get_template_context(
airflow/models/taskinstance.py:960: in _get_template_context
    dag_run=DagRunSDK.model_validate(dag_run, from_attributes=True),
E   pydantic_core._pydantic_core.ValidationError: 1 validation error for DagRun
E   logical_date
E     Input should be a valid datetime [type=datetime_type, input_value=None, input_type=NoneType]
E       For further information visit https://errors.pydantic.dev/2.10/v/datetime_type

It's curious/a massive oversight in tests that this is broken but it's only a provider test that picks it up!

@potiuk
Copy link
Member

potiuk commented Feb 6, 2025

It's curious/a massive oversight in tests that this is broken but it's only a provider test that picks it up!

Yep. I would really be curious to see what's the solution - maybe it's just a test case, maybe that's something that it's missing in the unit tests. Or maybe we should bring back all provider tests to be run on core changes - they are disabled now but seems to be breaking things more often.

potiuk added a commit to potiuk/airflow that referenced this pull request Feb 6, 2025
The apache#46295 removed the unique constraints but the case when logical
date is None is not handled.

This one fixes the problem but likely needs some unit tests and
fixes in the core
insomnes pushed a commit to insomnes/airflow that referenced this pull request Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-83 Remove Execution Date Unique Constraint from DAG Run area:db-migrations PRs with DB migration kind:documentation
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

8 participants