Skip to content

ES|QL support (#194) #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 21, 2025
Merged

Conversation

mashhurs
Copy link
Contributor

@mashhurs mashhurs commented Jul 18, 2025

Cherry pick of 5e3c464

  • ESQL and DSL executors are introduced. param can accept ES|QL query shape now. is introduced for initial step but needs team's feedback. DSL logics moved into DSL executors.

  • Apply suggestions from code review

Separate DSL and ESQL interface in the client.

  • Rebase against upstream main after target support added. Separate unit test for DSL. Address comments: do not save ES version in client, add apply target method in executors, set to target if target is defined, docs update.

  • Introduce query_type option which accepts dsl or esql to define a query shape. Remove multi-depth nested named_params and keep only top-level query_params which aligns with placeholder structure in the ES|QL.

  • Separate event referenced and static valued fields at initialization of the ESQL executor.

  • query_params now supports both Array and Hash types.

  • Add tech preview section under ESQL.

  • Place the query results based on the target specified. If not specified, first result will be set to event's top level.

  • Apply suggestions from code review

Doc corrections.

  • ES|QL result mapping to event doc correction.

  • Integration tests to run with credentials enabled and SSL configs.


(cherry picked from commit 5e3c464)

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

@mashhurs
Copy link
Contributor Author

I have bit-by-bit compared this change (especially lib/logstash/filters/elasticsearch.rb) to #194 and seems identical!

Failed CI step (INTEGRATION=true SNAPSHOT=true LOG_LEVEL=info ELASTIC_STACK_VERSION=8.future) is as expected:

#3 [elasticsearch internal] load metadata for docker.elastic.co/elasticsearch/elasticsearch:8.future
#3 ERROR: docker.elastic.co/elasticsearch/elasticsearch:8.future: not found
#4 [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future
#4 ERROR: docker.elastic.co/logstash/logstash:8.future: not found
------
 > [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future:

@mashhurs mashhurs requested a review from jsvd July 18, 2025 17:56
* ESQL and DSL executors are introduced.  param can accept ES|QL query shape now.  is introduced for initial step but needs team's feedback. DSL logics moved into DSL executors.

* Apply suggestions from code review

Separate DSL and ESQL interface in the client.

Co-authored-by: Rye Biesemeyer <[email protected]>

* Rebase against upstream main after target support added. Separate unit test for DSL. Address comments: do not save ES version in client, add apply target method in executors, set to target if target is defined, docs update.

Co-authored-by: Rye Biesemeyer <[email protected]>

* Introduce query_type option which accepts dsl  or esql to define a query shape. Remove multi-depth nested named_params and keep only top-level query_params which aligns with placeholder structure in the ES|QL.

* Separate event referenced and static valued fields at initialization of the ESQL executor.

* query_params now supports both Array and Hash types.

* Add tech preview section under ESQL.

* Place the query results based on the target specified. If not specified, first result will be set to event's top level.

* Apply suggestions from code review

Doc corrections.

Co-authored-by: João Duarte <[email protected]>

* ES|QL result mapping to event doc correction.

* Integration tests to run with credentials enabled and SSL configs.

---------

Co-authored-by: Rye Biesemeyer <[email protected]>
Co-authored-by: João Duarte <[email protected]>
(cherry picked from commit 5e3c464)
@mashhurs
Copy link
Contributor Author

I have bit-by-bit compared this change (especially lib/logstash/filters/elasticsearch.rb) to #194 and seems identical!

Failed CI step (INTEGRATION=true SNAPSHOT=true LOG_LEVEL=info ELASTIC_STACK_VERSION=8.future) is as expected:

#3 [elasticsearch internal] load metadata for docker.elastic.co/elasticsearch/elasticsearch:8.future
#3 ERROR: docker.elastic.co/elasticsearch/elasticsearch:8.future: not found
#4 [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future
#4 ERROR: docker.elastic.co/logstash/logstash:8.future: not found
------
 > [logstash internal] load metadata for docker.elastic.co/logstash/logstash:8.future:

Solved by #200

@mashhurs mashhurs merged commit 3964b91 into logstash-plugins:3.x Jul 21, 2025
3 checks passed
@mashhurs mashhurs deleted the esql-support-3.x branch July 21, 2025 16:15
@mashhurs
Copy link
Contributor Author

Published with v3.19.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants