Conversation
1c8cd2e to
6d6d8c4
Compare
|
Rebased onto latest main; all checks are green. This adds a first-class |
apache#73286 made every connection id an operator accepts a template field, and added the check-conn-id-templated hook to enforce it; QdrantIngestOperator gained conn_id in that change. QdrantSearchOperator accepted conn_id without templating it, so the hook fails on this branch once it is rebased onto main. Add it last, matching QdrantIngestOperator.
The test documents which fields users are expected to template, and conn_id is now one of them. Without it in the expected set, dropping conn_id from template_fields would be caught only by the static check, not by the unit tests.
6d6d8c4 to
c05a38f
Compare
|
Rebased onto main. #73286 has since made every connection id an operator accepts a template field, and added a Re-checked after the rebase: the qdrant unit tests pass against |
|
Hello @YAshhh29 - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 6 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
The Qdrant provider today lets users write vectors into a collection
(
QdrantIngestOperator) but has no operator for the other half of a RAGpipeline: reading them back out. Users who want to run a similarity search
from a DAG have to reach into
hook.conn.query_pointsdirectly and rememberto convert the returned pydantic
ScoredPointobjects into plain dicts soAirflow can serialize them to XCom -- a footgun that shows up as a cryptic
serialization error at runtime.
This PR adds
QdrantSearchOperator, a first-class task that closes thatgap. Every other vector-DB provider (Pinecone, Weaviate) is missing the
same operator; Qdrant is the leanest of the three (its hook doesn't even
have a
searchmethod today), so it's the cleanest place to start.How I found and verified this gap
This isn't tied to an existing issue -- I discovered it by auditing the
operator surface of every AI/ML provider in Airflow (openai, cohere,
pinecone, weaviate, qdrant, pgvector), the same audit approach behind
#69408 and #69534. For each provider I compared what the hook/client can
do to what's actually exposed as operators.
The pattern that jumped out: every vector-DB provider is missing a
search operator. Users can ingest with a proper operator but must fall
back to raw hook calls to query. That's the retrieval half of RAG living
outside the Airflow abstraction.
Before writing a line of code I confirmed:
0 open issues mentioning "qdrant search" -- greenfield, no one else
was building this.
qdrant-client 1.18.0(the provider pins
>=1.17.1) exposesquery_pointswith every oneof the 9 named parameters this operator forwards; the older
search()method is deprecated and slated for removal.
QueryResponse.pointsis
List[ScoredPoint], andScoredPoint.model_dump()produces theid/score/payload/vector/version/shard_key/order_valuedict shapethe operator promises callers.
python-modulesinprovider.yamlcovers any class inoperators/qdrant.py, so adding one needs zero registry edits.Only then did I write the code, in the small incremental steps you can
see in the six commits (hook -> hook tests -> operator -> operator tests
-> example DAG -> docs).
Design decisions
QdrantHook.search()wrapsQdrantClient.query_pointsand convertseach returned
ScoredPointto a plaindictviamodel_dump(). Theoperator is a ~10-line delegate on top. This mirrors the operator/hook
split every other provider uses -- and gives tests a clean seam to
mock at.
list[dict[str, Any]](id, score, payload, and optionally vector), so results land in XCom
without any user-side workaround.
query_points, not the deprecatedsearch(). Thesearch()API in
qdrant-clientis scheduled for removal in a future major;query_pointsis the modern surface (also supports named/sparsevectors, hybrid search, etc.). A regression test asserts we never
fall back to the deprecated method.
**kwargspassthrough. Forwards anyquery_pointsparameter wedon't enumerate (
using,prefetch,lookup_from, ...) so the hookstays forward-compatible with hybrid search and named vectors without
a follow-up PR.
What changes
providers/qdrant/src/airflow/providers/qdrant/hooks/qdrant.pyQdrantHook.search(...)method wrappingquery_points, returninglist[dict]viaScoredPoint.model_dump().providers/qdrant/src/airflow/providers/qdrant/operators/qdrant.pyQdrantSearchOperatorclass alongside the existingQdrantIngestOperator.template_fieldsincludecollection_name,query,query_filter,limitso a RAG DAG can XCom-pull a queryvector from an upstream embedding task.
providers/qdrant/tests/unit/qdrant/hooks/test_qdrant.pylist[dict]viamodel_dump; usesquery_points(not deprecatedsearch) with all named argsforwarded; extra
**kwargsalso forwarded.providers/qdrant/tests/unit/qdrant/operators/test_qdrant.pyexpected; every optional arg reaches the hook; template_fields
cover the runtime parameters; default
conn_idmatches the hook's.providers/qdrant/tests/system/qdrant/example_dag_qdrant.pyQdrantSearchOperatortask downstream of the existingingest task with
# [START/END] howto_operator_qdrant_searchmarkers.
providers/qdrant/docs/operators/qdrant.rst.. _howto/operator:QdrantSearchOperator:anchor and an
.. exampleinclude::pulling the DAG snippet.No
provider.yaml/get_provider_info.pychanges needed: the registrylists python-modules, not classes, so a new class in an existing module is
picked up automatically. No changelog edit either -- provider changelogs
are regenerated from
git logby the release manager perAGENTS.md.Testing
a standalone harness that runs the real hook/operator code with mocked
Qdrant client + a
BaseHook/BaseOperatorshim (full Airflow can'trun on Windows).
query_pointsaccepts every one of the 9 named parameters we forward, and
QueryResponse.pointsis aList[ScoredPoint]with the expectedmodel_dump()shape (id,score,payload,vector, ...).QdrantIngestOperatorstill constructs andbehaves identically -- we only added to the module, no existing code
was touched.
ruff check+ruff format --check: 26 files clean..github/instructions/code-review.instructions.md-- no red flags (no
time.time, noassertin prod, no newAirflowException, no British spellings, no missing tests).Was generative AI tooling used to co-author this PR?
Generated-by: GitHub Copilot (Claude Opus 4.6) following the guidelines