kubedns not auto-discoverable by Datadog

[mjpitz]

RKE version:

$ rke -v
rke version v0.2.2

By default, DataDog looks for the kubedns metrics on 10055 as per the comment here: DataDog/datadog-agent#1514 (comment)

When looking into the RKE setup of DNS, it binds the metrics port to be 10054 instead of their default port. This unfortunately causes a good number of valuable metrics to be missed by the collectors. Please add the following annotations to the kube-dns pod to enable auto-discovery by datadog:

metadata:
  annotations:
    service-discovery.datadoghq.com/kubedns.check_names: '["kube_dns"]'
    service-discovery.datadoghq.com/kubedns.init_configs: '[{}]'
    service-discovery.datadoghq.com/kubedns.instances: '[[{"prometheus_endpoint":"http://%%host%%:10055/metrics", "tags":["dns-pod:%%host%%"]}]]'

Reference Documentation Here:
https://github.com/DataDog/integrations-core/tree/master/kube_dns

gz#14409

[mjpitz]

Looking through the configuration more. It looks like DNS is configured with the proper port, it's just missing the auto-discovery annotations.

[deniseschannon]

instead of adding these annotations on the kube-dns pod directly for all RKE clusters, we should see if we can make it so that you can add annotations onto the kubedns pod through cluster.yml

[spolspol]

Same issue on AWS EKS with 'coredns' deployment that comes with.

[ AGENT ] 2020-02-21 11:13:39 UTC | ERROR | (runner.go:289 in work) | Error running check coredns: [{"message": "HTTPConnectionPool(host='10.200.4.104', port=9153): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe31bcbd310>: Failed to establish a new connection: [Errno 111] Connection refused',))", "traceback": "Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py", line 387, in run
self.check(copy.deepcopy(self.instances[0]))
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 45, in check
self.process(scraper_config)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 294, in process
for metric in self.scrape_metrics(scraper_config):
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 262, in scrape_metrics
response = self.poll(scraper_config)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 411, in poll
response = self.send_request(endpoint, scraper_config, headers)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 474, in send_request
cert=cert, verify=verify, auth=auth)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='10.200.4.104', port=9153): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe31bcbd310>: Failed to establish a new connection: [Errno 111] Connection refused',))
"}]

[nayrnet]

Being able to define custom service annotations via cluster.yml would be very nice to have w/RKE.

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.