Agent not honoring DD_COLLECT_EC2_TAGS when ec2 integration is turned on

[JohnlNguyen]

Additional environment details (Operating System, Cloud provider, etc):
datadog/agent:6.11.3
datadog/cluster-agent:1.3.1

Steps to reproduce the issue:

1. Set DD_COLLECT_EC2_TAGS: false
2. Turn on ec2 integration https://docs.datadoghq.com/integrations/amazon_ec2/
3. dd-agent still collects ec2 tags in metrics

Describe the results you received:
Metrics have ec2 tags
Describe the results you expected:
Metrics should not have ec2 tags

Additional information you deem important (e.g. issue happens only occasionally):

DD agent should honor the DD_COLLECT_EC2_TAGS rather than requiring users to modify the IAM policy to disable ec2:DescribeInstances in order to stop collecting ec2 tags

[semoac]

Same problem. We are trying to use dogstatsd but the custom metrics count just exploded because of this. (version 7.21.0 BTW)

[2rs2ts]

I had a support ticket open about our events having a bunch of garbage tags for the past few months. Forgot about this known issue. I have linked support to it. Thank you @semoac for commenting, this might help get a solution actually rolling.

[grosser]

@JohnlNguyen can you elaborate on where you did the "modify the IAM policy to disable ec2:DescribeInstances" (instance policy ? .. or permissions for the agent ... or something else ?) I'm trying to do the same to fix the same issue 😞

[stefansedich]

Is there any update from anyone at DataDog on this? I have tried to cover this with support a number of times but have not had any luck getting a solution.

[2rs2ts]

Is there any update from anyone at DataDog on this?

Sort of. I apologize for not circling back here to comment on the issue about what a DataDog lead solutions engineer told me in Jan 2021 before closing out my ticket.

I'm reaching out to follow up on this ticket thread, after a period of inactivity. Firstly, I'd like to apologize for the silence on our end here. After performing an audit on our side and reviewing your case in more detail, I can see that this bug / issue has been triaged by our product and engineering teams to be worked on, but I don't yet have a firm ETA for when this will be completed.

So, it's in their backlog but not roadmapped, most likely. I won't share the name of that lead solutions engineer publicly for their privacy's sake. What I will do is make a follow-up ticket saying "hey, don't forget about us!" Maybe the fact that my past self got far enough to get a lead solutions engineer involved will have more "weight" with them and they'll bump up the priority on fixing it... but IME every time I report a bug, it takes them years to fix it, if they fix it at all. Of course, I seem to only report the really tricky bugs that take sometimes weeks of discussion to get them to nail it down, so, take that with a grain of salt. I'm not trying to shame them too much, just a tiny bit 🤏

I'm bad at remembering to follow up with GitHub threads that aren't active and showing up in my notifications a lot, so don't expect a response from me unless you ping me in a couple weeks. And definitely don't get your hopes up that I'll have good news. Nonetheless, I'm still going to try. Edit: good news, they replied basically immediately. Here is their response:

The team appreciates you reaching out about the issue. Please know the team is still working on your case regarding the issue in development as of today 3/31/23.

After performing an audit on our side and reviewing your case in more detail, I can see that this bug / issue is being triaged by our product and engineering teams , but I don't yet have a firm ETA for when this will be completed.

That being said, I'm going to close out this ticket thread for now, but if this is an urgent issue for you and your team please feel free to reply here and I'd be happy to discuss your case with our product / engineering teams in more detail.

So in other words, I would have to escalate it as an urgent issue to get faster/any progress. It's not urgent for me, so I don't want to lie to them. It's a bummer, but it seems like there's been no progress since over 2 years ago.

[stefansedich]

Thanks @2rs2ts appreciate the update, and I totally understand about missing notifications on old GitHub threads!

[bunnybilou]

Invested time last Friday attempting to configure the settings correctly, but encountered issues even with the latest version of the agent.

What's puzzling is that in the host page, the tag is only configured in the AWS section, not in the Datadog section. However, tags are still being set in metrics/logs.

[bunnybilou]

Hey, any updates on this issue?

Removing ec2:DescribeInstances is helping here but all the instance metrics are missing in Datadog because of this.

I want to monitor the disk usage of one of our instances, but I'm unable to do so because I've removed the ec2:DescribeInstances action from the policy.

[2rs2ts]

@bunnybilou if you haven't opened a ticket with datadog support, please do so and mention this github issue. It's a real pain and I'm getting frustrated that it's still not fixed, too.