AIP-84 Limit and filters not applied correctly for dag runs API.

[tirkarthi]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

It seems the limit filter is not honored in the dagruns api as below. Though limit 14 is applied all dagruns are returned. This is slightly tricky to reproduce since the limit is passed correctly to API and doesn't work through UI but curl works. Ref #46504 (comment)

curl http://localhost:8000/public/dags/tutorial_taskflow_api/dagRuns\?limit\=14

What you think should happen instead?

No response

How to reproduce

1. Visit the dags page with more than 14 runs and the filter is not honored.
2. curl http://localhost:8000/public/dags/tutorial_taskflow_api/dagRuns\?limit\=14

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[insomnes]

From your initial PR conversation, I understood you are referencing the part where you extract total_entries from the response, and it shows all total_entries despite the limit set to 14.

If so I believe that this number is not affected by pagination limits and offsets, the total_entries would be based on count from filters= fields.

    dag_run_select, total_entries = paginated_select(
        statement=query,
        filters=[logical_date, start_date_range, end_date_range, update_at_range, state],
        order_by=order_by,
        offset=offset,
        limit=limit,
        session=session,
    )

And with curl it works as you mention:

curl run for 2 entries

curl -X 'GET' \
      'http://localhost:29091/public/dags/example_bash_decorator/dagRuns?limit=2&order_by=-start_date' \
      -H 'accept: application/json' | jq
{
"dag_runs": [
  {
    "dag_run_id": "manual__2025-02-08T23:20:51.665298+00:00",
    "dag_id": "example_bash_decorator",
    "logical_date": "2025-02-08T23:20:51.673703Z",
    "queued_at": "2025-02-08T23:20:51.688648Z",
    "start_date": "2025-02-08T23:38:06.692506Z",
    "end_date": "2025-02-08T23:38:09.660110Z",
    "data_interval_start": "2025-02-08T23:20:51.673703Z",
    "data_interval_end": "2025-02-08T23:20:51.673703Z",
    "last_scheduling_decision": "2025-02-08T23:38:09.656040Z",
    "run_type": "manual",
    "state": "failed",
    "external_trigger": true,
    "triggered_by": "rest_api",
    "conf": {},
    "note": null
  },
  {
    "dag_run_id": "manual__2025-02-08T23:04:32.759689+00:00",
    "dag_id": "example_bash_decorator",
    "logical_date": "2025-02-08T23:04:32.759689Z",
    "queued_at": "2025-02-08T23:04:32.784971Z",
    "start_date": "2025-02-08T23:04:33.602888Z",
    "end_date": "2025-02-08T23:04:36.819802Z",
    "data_interval_start": "2025-02-08T23:04:32.759689Z",
    "data_interval_end": "2025-02-08T23:04:32.759689Z",
    "last_scheduling_decision": "2025-02-08T23:04:36.817639Z",
    "run_type": "manual",
    "state": "failed",
    "external_trigger": true,
    "triggered_by": "ui",
    "conf": {},
    "note": null
  }
],
"total_entries": 127
}

Or the problem is that API returns the wrong number of entries in dag_runs field?

[pierrejeambrun]

I have identified the problem.

I should be able to open a PR to fix that.

[tirkarthi]

Thanks @pierrejeambrun , from my understanding the QueryLimit and QueryOffset are module level variable initialised only once and using it on type annotations the same LimitFilter object gets used for requests. This is reproducible only in UI with parallel requests where the limit values of different APIs overwrite each other and the final value is taken into account. In curl since it's sequential the limit filter object is updated properly. This might be applicable to other filters as well constructed in similar manner.

[insomnes]

Sorry for my previous question, but the exact problem was initially unclear to me.
Yesterday I was able to reproduce the behavior you described now with async requests to API with a single HTTP session, but didn't have time to explain what I found.

Great to see that the root cause was identified already by Pierre!

[pierrejeambrun]

PR: #46619