Sending DocumentStore-specific filters without LogicalFilterClause parsing

[denlogv]

Hello,

the REST API provided by Haystack has its own syntax for defining metadata filters. It is okay for simple filtering, but sometimes seems to fall short in mirroring the actual filtering abilities of the DocumentStore in use.

Example:
We use the OpenSearchDocumentStore and want a way to define a wildcard query, for example, to filter filenames that contain "invoice" in them. That's how it would look in OpenSearch:
{ "wildcard" : { "content" : { "value" : "*invoice*" } } }

Is there a way to define such a query using Haystack without defining a new operator for wildcard? And if not, what is the recommended way to surpass the Haystack filtering syntax and just send an OpenSearch filter? Currently it looks like one of the options would be to subclass the Retriever and override its retrieve method to accept custom_query as a parameter (because now it looks like the custom_query template is created during initialization and cannot be dynamically changed).

[ZanSara]

Hey @denlogv! If I understood your problem correctly, it might just have been fixed with release 1.20. Check out the release notes: https://github.com/deepset-ai/haystack/releases/tag/v1.20.0 Under New Features you should see an entry for the latest changes to how we handle custom_query, and it now allows custom filters to be sent directly.

Therefore if you define the correct custom_query at init time (as shown in the code snippets of the release notes) your filters will be sent unchanged to the docstore and you should be able to use Opensearch-specific filter syntax.

Let us know if that solves your problem

[denlogv]

Hey @ZanSara, if I understand the code correctly this is actually not the case. Here is the method SearchEngine._construct_query_body, which is called in OpenSearchDocumentStore.query:

https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/search_engine.py#L1085

Here the filters go through LogicalFilterClause.parse(filters or {}).convert_to_elasticsearch() which makes them use the Haystack filtering syntax. I also checked the parse method if it allows not having Haystack filters and I don't think it does. The filters will go one by one through ComparisonOperation.parse and will output an empty list:

https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/filter_utils.py#L123

[denlogv]

I think it would be ideal if one could pass the custom_query in each invocation of the retrieve method, because currently you allow other fields, such as if top_k, index, scale_score, all_terms_must_match to do so:

https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/retriever/base.py#L34
https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/retriever/sparse.py#L225

Also, it looks like the EmbeddingRetriever doesn't have support for custom_queries at all?

https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/opensearch.py#L800

[sanjayc2]

Hello, I would like to follow up on @denlogv request for a wildcard (custom) query, since I've not been able to find how do it in latest version of Haystack. As noted, Haystack has its own syntax for defining metadata filters. The current filtering mechanisms seem to fall short in mirroring the actual filtering abilities of some of the supported DocumentStores (e.g., Milvus has the "LIKE" operator, and Qdrant has a "full text match" filter condition, while Haystack only has the IN operator, which does not work in this situation). I too would need this functionality for an embedding retriever, so that each invocation retrieves a set of chunks filtered by the condition specified in a custom query. I would appreciate it if someone can point me to how to accomplish a similar "LIKE" or wildcard (i.e., {"value": "*invoice*"} ) operation in @denlogv request. And if that is still not possible, could I raise this request again. Thank you in advance. P.S. I've only been working with Haystack a couple months, and I find it far better designed than Langchain (the other popular app).