-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add blog post about the OTTL context inference feature #6290
base: main
Are you sure you want to change the base?
Changes from 2 commits
eaceb27
03524b8
3236c44
77c60fb
95b5b2e
4e75d4a
17fe023
241b8bd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
--- | ||
title: OTTL contexts just got easier with context inference | ||
linkTitle: OTTL contexts just got easier | ||
date: 2025-02-17 | ||
author: '[Edmo Vamerlatti Costa](https://github.com/edmocosta) (Elastic)' | ||
draft: true # TODO: remove this line once your post is ready to be published | ||
issue: 6289 | ||
sig: Collector SIG | ||
cSpell:ignore: OTTL Vamerlatti | ||
--- | ||
|
||
Selecting the right context for running OTTL statements can be challenging, even | ||
for experienced users. Choosing the correct context impacts both accuracy and | ||
efficiency, as using higher-level contexts can avoid unnecessary iterations | ||
through nested lower-level contexts. | ||
|
||
To simplify this process, the OpenTelemetry community is excited to announce the | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
OTTL | ||
[context inference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#context-inference) | ||
support for the | ||
[transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor). | ||
This feature removes the need to manually specify contexts, improving statement | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
processing efficiency by automatically selecting the most appropriate one. This | ||
optimization ensures that data transformations are both accurate and performant. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not strictly necessary to mention this, and we touch on it when talking about the flat configuration style, but from a general UX standpoint, this change also relieves users from needing to know the concept of contexts and instead just think about the data they want to work with. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we should highlight this point as it is the primary motivator for this work. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we also point out that soon this will make statements portable between components that use OTTL? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It's kind of mentioning it here, but without providing much details. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I've pushed an extra change mentioning that. Could you please take a look again? thanks! |
||
|
||
## How it works? | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Starting with version `0.120.0`, the transform processor supports two new | ||
[context-inferred configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#context-inferred-configurations) | ||
styles. The first one closely resembles the existing configuration format, while | ||
the second offers a simpler and flatter approach. | ||
|
||
### Structured configuration | ||
|
||
The context-inferred structured configuration style closely resembles the | ||
existing format. For example, consider the following configuration: | ||
|
||
```yaml | ||
metric_statements: | ||
- context: datapoint | ||
statements: | ||
- set(metric.description, "counter") where attributes["my.attr"] == "some" | ||
``` | ||
|
||
It can now be written as: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- statements: | ||
- set(metric.description, "counter") where datapoint.attributes["my.attr"] == "some" | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
In this example, the `context` value is suppressed and is automatically inferred | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
to `datapoint`, as it is the only context present in the statements that | ||
supports parsing both`datapoint` and `metric` data. | ||
|
||
All paths in the statements must be prefixed with their respective contexts. | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
These prefixes are required for context-inferred configurations and serve as | ||
hints for selecting the best match. It also makes statements unambiguous and | ||
portable between components. | ||
|
||
If we update the above configuration removing the `where` condition: | ||
|
||
```yaml | ||
metric_statements: | ||
- statements: | ||
- set(metric.description, "counter") | ||
``` | ||
|
||
The context inferrer would select the `metric` context instead, since | ||
`datapoint` is no longer required, and although it would be possible to run the | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
statements using the`datapoint` context, `metric` is the most efficient option. | ||
|
||
### Flat configuration | ||
|
||
The flat configuration style simplifies the configuration by allowing users to | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
list all statements together, without worrying about contexts or extra | ||
configuration structures. This style support statements from multiple contexts | ||
and does not require grouping them separately. | ||
|
||
To illustrate this, compare the following configuration: | ||
|
||
```yaml | ||
metric_statements: | ||
- context: resource | ||
statements: | ||
- keep_keys(attributes, ["host.name"]) | ||
- context: metric | ||
statements: | ||
- set(description, "Sum") where type == "Sum" | ||
- convert_sum_to_gauge() where name == "system.processes.count" | ||
- context: datapoint | ||
statements: | ||
- limit(attributes, 100, ["host.name"]) | ||
``` | ||
|
||
With the new flat style configuration format, the same logic is expressed more | ||
edmocosta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
concisely: | ||
|
||
```yaml | ||
metric_statements: | ||
- keep_keys(resource.attributes, ["host.name"]) | ||
- set(metric.description, "Sum") where metric.type == "Sum" | ||
- convert_sum_to_gauge() where metric.name == "system.processes.count" | ||
- limit(datapoint.attributes, 100, ["host.name"]) | ||
``` | ||
|
||
This streamlined approach enhances readability and makes configuration more | ||
intuitive. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing here is probably a subsection stating when you should use which form. Users will generally want to know which they should generally prefer, and when. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried to explain that in each configuration section:
Do you think we need to provide more details on that? |
||
## Try it out | ||
|
||
As we wrap up, we encourage users to explore this new functionality and take | ||
advantage of its benefits in their telemetry pipelines! | ||
|
||
If you have any questions or suggestions, we’d love to hear from you! Join the | ||
conversation in the `#otel-collector` channel on the | ||
[CNCF Slack workspace](https://slack.cncf.io/). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll want a link for context here that disambiguates it from the "context propagation" context that is predominantly used in the site.