[Puppet] Datadog agent does not collect Apache logs

[dduportal]

While working on #3485 and #3494 , we found that the Datadog agent integration with Apache is incomplete: we do not have Apache log collection (only metrics).

• https://docs.datadoghq.com/integrations/apache/?tab=host#log-collection states that a section logs must be configured in the apache integration configuration file (/etc/datadog-agent/conf.d/apache.d/conf.yaml)

• The main cause is the Puppet module for datadog agent does not support all the features of the integrations: Logs still treated as an integration DataDog/puppet-datadog-agent#612 (comment)

  • A first try to the proposed "hack" shows it's a cause of a lot of spaghetti puppet code: "if apachemisc profile is applied, then add the apache integration with custom elements in it"
  • Tried with success a static configuration file which only depends of the presence or absence of apache instead.

• Another blocker is the file /etc/datadog-agent/conf.d/process.yaml, related to https://docs.datadoghq.com/integrations/process/, which is empty and invalid, causing trouble for datadog-agent for monitoring the Apache (and other) processes.

  • Tried to delete it from Puppet and the filesystem, and then let datad-go manage with its default, until we'll need to specify it
  • Root cause is the datadogagent_check custom puppet profile we have: let's remove it

• The datadog-agent process need to read the log files for apache (at least?) so these files need the permission read for others (to avoid adding the agent to an adminisrtator group) and execute on the containgin directories

[dduportal]

First step in jenkins-infra/jenkins-infra#2764 deployed with success

Next tasks:

• Fix the warning messag surfacing, related to feat(datadog) define custom hostname for each VM and enable EC2 tags collection when needed jenkins-infra#2763 (comment)
• Remove the file /etc/datadog-agent/conf.d/process.yaml on all VMs
• set the permissions on the /var/log/apache2 directory and ensure it's tracked by Puppet

[dduportal]

• fix(datadog-agent) remove ec2 tags collection as we do not use IMDB jenkins-infra#2767 to remove ec2 tags
• Removed the file with

for machine in ci.jenkins.io cert.ci.jenkins.io archives.jenkins.io edamame.jenkins.io lettuce.jenkins.io census.jenkins.io usage.jenkins.io trusted.ci.jenkins.io trusted-agent-1 pkg.origin.jenkins.io vpn.jenkins.io private.vpn.jenkins.io bounce puppet.jenkins.io; do echo "=== Machine: ${machine}";ssh "${machine}" sudo rm -f /etc/datadog-agent/conf.d/process.yaml;done

• Apache access for datadog agent:
  • feat(apache|datadog-agent) ensure that datadog-agent can read apache2 logs directory jenkins-infra#2765
  • fixup in hotfix: add ruby puppet module jenkins-infra#2766

[dduportal]

https://github.com/jenkins-infra/puppet-apachelogcompressor is now archived

[dduportal]

Apache Dashboard in datadog now shows all VMs 🥳

We can see the log collection working as expected: