Skip to content

AIP-76: propagate partition_date to consumers #67239

Description

@nathadfield

Description

Propagate partition_date (datetime) from the producer's DagRun to the consumer DagRun and Context for partitioned asset events (AIP-76), so that templates can use the existing Airflow filter idiom {{ partition_date | ds }} / {{ partition_date | ds_nodash }} instead of parsing partition_key by hand.

Use case/motivation

After #65340 / #65359, consumers can access partition_key (a string) directly from Context. But the partition's underlying datetime, partition_date, is None on consumer DagRuns. The column exists on DagRun (added via #61167) and is populated on the producer side, but nothing carries it across the asset event boundary.

This forces every consumer of a partitioned asset to parse the key string manually. With the default CronPartitionTimetable key_format of %Y-%m-%dT%H:%M:%S, the workarounds look like:

# Inline string slicing - opaque, depends on key_format
WHERE dt = "{{ partition_key[:10] }}"

# Or a userland plugin macro
WHERE dt = "{{ macros.partition_ds(dag_run) }}"

Both push date-parsing back to userland, which is at odds with the project's canonical filter-based templating (per the templates ref):

-- Non-partitioned DAG today
WHERE dt = "{{ logical_date | ds }}"

-- Partitioned-asset consumer should be able to write
WHERE dt = "{{ partition_date | ds }}"

AIP-76 is the headline scheduling feature of 3.2; the templating story should match the rest of Airflow's datetime ergonomics.

Proposal

When materialising a consumer DagRun from a partitioned asset event, populate DagRun.partition_date from the event's partition (the producer's partition_date, or re-derived from partition_key + key_format). Surface it in Context alongside partition_key, the same way #65359 surfaced the string:

ctx = get_current_context()
ctx["partition_date"]                   # datetime
ctx["partition_key"]                    # str (already in Context)

Templates then use the filter form that already works everywhere else:

WHERE dt = "{{ partition_date | ds }}"
WHERE dt_nodash = "{{ partition_date | ds_nodash }}"
WHERE ts = "{{ partition_date | ts_nodash }}"

No new macros, no new naming - just extending the existing partition_date column's reach to the consumer side and the same Context-exposure pattern #65359 established for partition_key.

Counterpoint

CronPartitionTimetable.key_format is configurable; not every partition key is a date. The existing DagRun.partition_date column is already nullable=True for this reason. The same gate should apply on the consumer side: partition_date is populated only when the producer's partition is date-shaped. For non-date partition keys (e.g. region codes, run IDs), partition_date stays None and users fall back to partition_key. This matches the existing producer-side behaviour.

Related issues

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions