Skip to content

Commit bd81062

Browse files
AntonEliatrakolchfa-awsnatebower
authored
Adding example to contains data prepper (#11356)
* adding example to contains function Signed-off-by: Anton Rubin <[email protected]> * adding example to contains function Signed-off-by: Anton Rubin <[email protected]> * Update contains.md Signed-off-by: AntonEliatra <[email protected]> * Update _data-prepper/pipelines/contains.md Signed-off-by: kolchfa-aws <[email protected]> * Update _data-prepper/pipelines/contains.md Signed-off-by: Nathan Bower <[email protected]> --------- Signed-off-by: Anton Rubin <[email protected]> Signed-off-by: AntonEliatra <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Nathan Bower <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
1 parent 70d2ba5 commit bd81062

File tree

1 file changed

+80
-2
lines changed

1 file changed

+80
-2
lines changed

_data-prepper/pipelines/contains.md

Lines changed: 80 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ The function returns `true` if the substring specified in the second argument is
1818
For example, if you want to check if the string `"abcd"` is contained within the value of a field named `message`, you can use the `contains()` function as follows:
1919

2020
```
21-
contains('/message', 'abcd')
21+
'contains(/message, "abcd")'
2222
```
2323
{% include copy.html %}
2424

@@ -27,11 +27,89 @@ This call returns `true` if the field `message` contains the substring `abcd` or
2727
Alternatively, you can use a literal string as the first argument:
2828

2929
```
30-
contains('This is a test message', 'test')
30+
'contains("This is a test message", "test")'
3131
```
3232
{% include copy.html %}
3333

3434
In this case, the function returns `true` because the substring `test` is present within the string `This is a test message`.
3535

3636
The `contains()` function performs a case-sensitive search.
3737
{: .note}
38+
39+
## Example
40+
41+
The following pipeline uses the `contains()` function to add a Boolean flag `has_test` based on a substring in `/message` and filters out non-matching events, forwarding only messages that contain the string `ERROR` to OpenSearch:
42+
43+
```yaml
44+
contains-demo-pipeline:
45+
source:
46+
http:
47+
ssl: false
48+
49+
processor:
50+
- add_entries:
51+
entries:
52+
- key: has_test
53+
value_expression: contains(/message, "test")
54+
- drop_events:
55+
drop_when: not contains(/message, "ERROR")
56+
57+
sink:
58+
- opensearch:
59+
hosts: ["https://opensearch:9200"]
60+
insecure: true
61+
username: admin
62+
password: admin_password
63+
index_type: custom
64+
index: demo-index-%{yyyy.MM.dd}
65+
```
66+
{% include copy.html %}
67+
68+
You can test the pipeline using the following command:
69+
70+
```bash
71+
curl -sS -X POST "http://localhost:2021/log/ingest" \
72+
-H "Content-Type: application/json" \
73+
-d '[
74+
{"message":"ok hello"},
75+
{"message":"this has test but ok"},
76+
{"message":"ERROR: something bad"},
77+
{"message":"ERROR: unit test failed"}
78+
]'
79+
```
80+
{% include copy.html %}
81+
82+
The documents stored in OpenSearch contain the following information:
83+
84+
```json
85+
{
86+
...
87+
"hits": {
88+
"total": {
89+
"value": 2,
90+
"relation": "eq"
91+
},
92+
"max_score": 1,
93+
"hits": [
94+
{
95+
"_index": "demo-index-2025.10.21",
96+
"_id": "5YACB5oBqZitdAAb4n3r",
97+
"_score": 1,
98+
"_source": {
99+
"message": "ERROR: something bad",
100+
"has_test": false
101+
}
102+
},
103+
{
104+
"_index": "demo-index-2025.10.21",
105+
"_id": "5oACB5oBqZitdAAb4n3r",
106+
"_score": 1,
107+
"_source": {
108+
"message": "ERROR: unit test failed",
109+
"has_test": true
110+
}
111+
}
112+
]
113+
}
114+
}
115+
```

0 commit comments

Comments
 (0)