Allow excluding EC2 tags (implement exclude_ec2_tags configuration option)

[moleskin-smile]

We're using collect_ec2_tags configuration option to tag our Datadog metrics and events with AWS EC2 Instance tags. It's an useful option to tag cloud resources and Datadog events and metrics in a consistent manner.

Not all tags are that useful. It's a default and common practice that EC2 instances are tagged with "Name" tags. The purpose of these tags is to help navigate through EC2 instances in AWS Console, but they don't have any semantic meaning in AWS.

The "Name" tag name is so generic it doesn't bring that much value attached to all metrics and events produced on EC2 instances. Moreover some of Datadog metrics and events contain actually useful "name" tags creating problems with dashboards and especially monitors (grouping). This is quite serious issue we can't deal with easily using Datadog in-app configuration.

Examples of tag collision:

• Kubernetes events (search for source:kubernetes name:* in Events explorer)
• external-secrets metrics gathered via openmetrics integration

We also can't easily remove these tags without sacrificing their usefulness in AWS Console and without maintaining a burden of having a forked Terraform module we use to provision Kubernetes nodes. My attempt of allowing Name tag removal was recently rejected: terraform-aws-modules/terraform-aws-eks#2229.

Unwanted tags could also affect custom metric costs which could in turn affect customer satisfaction negatively.

We'd like to have an ability to configure list of EC2 tags to exclude in Datadog agents (i.e. exclude_ec2_tags). "Name" tags are only one example of tag collision that can happen.

There's similar option for GCE – exclude_gce_tags.

(I've requested this via a support request already, but this is sort of a bug – we can't create monitors based on metrics and events mentioned above with grouping by name).

[alexb-img]

Hi @moleskin-smile do you think this is similar / the same as #3130 ?
If so it would add weight to both. thanks

[moleskin-smile]

Thanks for looking at this. It's definitely similar, but some of issues described there can be solved now with Metrics Without Limits. The tag collision can't.

[alexb-img]

Thanks, agree: We require tags to be filtered/removed at source(ingest) also. Metrics without limits only removes tags during indexing not ingest.

[hush-hush]

Hi @moleskin-smile,

A exclude_ec2_tags option will be added in the 7.42.0 version of the agent: #14344

The new option works in the same way as exclude_gce_tags.

Thanks for reporting this feature request !

[moleskin-smile]

Hi @hush-hush. That's awesome! Thank you.

[nijogeorgep]

@hush-hush, when can we expect the public release of 7.42.0? is it already released?

[hush-hush]

@nijogeorgep, 7.42.0 release process has not yet started. Every 6 weeks we start a new agent release that goes to an extensive QA phase. I can't tell you exactly when 7.42.0 will be out but we're aiming at mid January. Do keep in mind that it will be release "when ready", so that date is very much what we are aiming for not a commitment.

[hush-hush]

I'm closing this issue since Agent version 7.42.0 has been released.