-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add blog post about the OTTL context inference feature #6290
Open
edmocosta
wants to merge
8
commits into
open-telemetry:main
Choose a base branch
from
edmocosta:add-ottl-contexts-just-got-easier-blogpost
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
eaceb27
Add OTTL context inference blogpost
edmocosta 03524b8
Update refcache.json
edmocosta 3236c44
Apply suggestions from code review
edmocosta 77c60fb
Merge branch 'main' into add-ottl-contexts-just-got-easier-blogpost
edmocosta 95b5b2e
Fix format
edmocosta 4e75d4a
Changed to emphasizing the flat configuration style first
edmocosta 17fe023
Apply review suggestion
edmocosta 241b8bd
Apply suggestions from code review
edmocosta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,134 @@ | ||
--- | ||
title: OTTL contexts just got easier with context inference | ||
linkTitle: OTTL contexts just got easier | ||
date: 2025-02-17 | ||
author: '[Edmo Vamerlatti Costa](https://github.com/edmocosta) (Elastic)' | ||
draft: true # TODO: remove this line once your post is ready to be published | ||
issue: 6289 | ||
sig: Collector SIG | ||
cSpell:ignore: OTTL Vamerlatti | ||
--- | ||
|
||
Selecting the right OTTL context for running statements can be challenging, even | ||
for experienced users. Choosing the correct option impacts both accuracy and | ||
efficiency, as using higher-level OTTL contexts can avoid unnecessary iterations | ||
through nested lower-level contexts. | ||
|
||
To simplify this process, the OpenTelemetry community is excited to announce | ||
OTTL | ||
[context inference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#context-inference) | ||
support for the | ||
[transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor). | ||
This feature removes the need to manually specify OTTL contexts, improving | ||
statement processing efficiency by automatically selecting the most appropriate | ||
one. This optimization ensures that data transformations are both accurate and | ||
performant, allowing users to focus solely on their data without needing to | ||
understand the underlying concept of OTTL contexts. | ||
|
||
## How does it work? | ||
|
||
Starting with version `0.120.0`, the transform processor supports two new | ||
[context-inferred configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#context-inferred-configurations) | ||
styles. The first one offers a simpler and flatter approach, while the second | ||
closely resembles the existing configuration format. | ||
|
||
### Flat configuration | ||
|
||
The flat configuration style simplifies configuration by allowing users to list | ||
all statements together, without worrying about OTTL contexts or extra | ||
configuration structures. This style support statements from multiple OTTL | ||
contexts and does not require grouping them separately. | ||
|
||
To illustrate this, compare the following configuration: | ||
|
||
```yaml | ||
metric_statements: | ||
- context: resource | ||
statements: | ||
- keep_keys(attributes, ["host.name"]) | ||
- context: metric | ||
statements: | ||
- set(description, "Sum") where type == "Sum" | ||
- convert_sum_to_gauge() where name == "system.processes.count" | ||
- context: datapoint | ||
statements: | ||
- limit(attributes, 100, ["host.name"]) | ||
``` | ||
|
||
With the new flat configuration style, the same logic is expressed more | ||
concisely by simply providing a list of statements: | ||
|
||
```yaml | ||
metric_statements: | ||
- keep_keys(resource.attributes, ["host.name"]) | ||
- set(metric.description, "Sum") where metric.type == "Sum" | ||
- convert_sum_to_gauge() where metric.name == "system.processes.count" | ||
- limit(datapoint.attributes, 100, ["host.name"]) | ||
``` | ||
|
||
This streamlined approach enhances readability and makes configuration more | ||
intuitive. To use this flat configuration, all paths in the statements must be | ||
prefixed with their respective OTTL contexts. These prefixes are required for | ||
all context-inferred configurations and serve as hints for selecting the best | ||
match. It also makes statements unambiguous and portable between components | ||
using OTTL. | ||
|
||
### Structured configuration | ||
|
||
The context-inferred structured configuration style closely resembles the | ||
existing format and allows users to leverage the benefits of context inference | ||
while providing granular control over statement configurations, such as | ||
`error_mode` and `conditions`. For example, consider the following | ||
configuration: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- context: datapoint | ||
conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") where attributes["my.attr"] == "some" | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
The above can now be written as: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") where datapoint.attributes["my.attr"] == "some" | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
In this example, the `context` value is omitted and is automatically inferred to | ||
`datapoint`, as it is the only OTTL context present in the statements that | ||
supports parsing both `datapoint` and `metric` data. | ||
|
||
If we update the above configuration removing the `datapoint` usage: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
The context inferrer would select the `metric` OTTL context instead, since no | ||
data points are accessed. Although it would be possible to run the statements | ||
using the `datapoint` OTTL context, `metric` is the most efficient option. | ||
|
||
## Try it out | ||
|
||
As we wrap up, we encourage users to explore this new functionality and take | ||
advantage of its benefits in their telemetry pipelines! | ||
|
||
If you have any questions or suggestions, we’d love to hear from you! Join the | ||
conversation in the `#otel-collector` channel on the | ||
[CNCF Slack workspace](https://slack.cncf.io/). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing here is probably a subsection stating when you should use which form. Users will generally want to know which they should generally prefer, and when.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to explain that in each configuration section:
Do you think we need to provide more details on that?