There was an error querying the ntp host #1532

adamgotterer · 2018-03-27T18:47:25Z

Describe what happened:
Launched the agent and the logs look like:

[ AGENT ] 2018-03-27 18:42:06 UTC | INFO | (transaction.go:129 in Process) | Successfully posted payload to "https://6-1-0-app.agent.datadoghq.com/intake/?api_key=*************************370fe"
[ AGENT ] 2018-03-27 18:42:12 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:49888->45.76.244.202:123: i/o timeout
[ AGENT ] 2018-03-27 18:42:27 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:58747->198.137.202.56:123: i/o timeout
[ AGENT ] 2018-03-27 18:42:37 UTC | INFO | (serializer.go:196 in SendJSONToV1Intake) | Sent processes metadata payload, size: 410 bytes.
[ AGENT ] 2018-03-27 18:42:42 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:44762->38.229.71.1:123: i/o timeout
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check cpu
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check cpu
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check disk
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check disk
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check docker
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check docker
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check file_handle
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check file_handle
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check io
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check io
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check load
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check load
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check memory
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check memory
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check network
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:302 in work) | Done running check network
[ AGENT ] 2018-03-27 18:42:52 UTC | INFO | (runner.go:246 in work) | Running check ntp
[ AGENT ] 2018-03-27 18:42:57 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:56134->162.210.111.4:123: i/o timeout
[ AGENT ] 2018-03-27 18:42:57 UTC | INFO | (runner.go:302 in work) | Done running check ntp
[ AGENT ] 2018-03-27 18:42:57 UTC | INFO | (runner.go:246 in work) | Running check uptime
[ AGENT ] 2018-03-27 18:42:57 UTC | INFO | (runner.go:302 in work) | Done running check uptime
[ AGENT ] 2018-03-27 18:43:12 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:43058->162.210.111.4:123: i/o timeout
[ AGENT ] 2018-03-27 18:43:27 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:46164->107.161.29.207:123: i/o timeout
[ AGENT ] 2018-03-27 18:43:42 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:39099->162.210.111.4:123: i/o timeout
[ AGENT ] 2018-03-27 18:43:57 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:52643->171.66.97.126:123: i/o timeout
[ AGENT ] 2018-03-27 18:44:12 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:34427->162.210.111.4:123: i/o timeout
[ AGENT ] 2018-03-27 18:44:21 UTC | INFO | (transaction.go:129 in Process) | Successfully posted payload to "https://6-1-0-app.agent.datadoghq.com/api/v1/check_run?api_key=*************************370fe"
[ AGENT ] 2018-03-27 18:44:27 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:53556->204.11.201.10:123: i/o timeout
[ AGENT ] 2018-03-27 18:44:42 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:59500->172.98.193.44:123: i/o timeout
[ AGENT ] 2018-03-27 18:44:57 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:33902->96.244.96.19:123: i/o timeout
[ AGENT ] 2018-03-27 18:45:12 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 172.17.0.5:36223->198.137.202.56:123: i/o timeout

Additional environment details (Operating System, Cloud provider, etc):
Docker, AWS, Amazon Linux

The text was updated successfully, but these errors were encountered:

shine17 · 2018-04-14T06:21:28Z

am also facing the same issue . any update on this issue ? any datadog engineer working on this ? please prioritize this

2018-04-14 06:17:25 UTC | INFO | (runner.go:246 in work) | Running check ntp
2018-04-14 06:17:30 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 25.128.37.38:37261->216.218.220.101:123: i/o timeout
2018-04-14 06:17:30 UTC | INFO | (runner.go:302 in work) | Done running check ntp
2018-04-14 06:17:30 UTC | INFO | (runner.go:246 in work) | Running check uptime
2018-04-14 06:17:30 UTC | INFO | (runner.go:302 in work) | Done running check uptime
2018-04-14 06:17:45 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 25.128.37.38:49068->162.210.111.4:123: i/o timeout
2018-04-14 06:18:00 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 25.128.37.38:57120->216.218.220.101:123: i/o timeout

am using datadog docker image datadog/agent:6.1.2-jmx as a side car container in K8s

It seems because of this time sync issue , am not seeing the jmx metrics in datadog dashboards.

datadog-agent/pkg/collector/corechecks/network/ntp.go

Line 122 in 68f1076

log.Infof("There was an error querying the ntp host: %s", err)

shine17 · 2018-04-14T06:48:21Z

This is what i get when i do a datadog agent ntp check from the side car

root@something*********-hj2hx:/opt/datadog-agent/bin/agent# ./agent check ntp -r -l INFO
%!s(int=442840760) | INFO | (tagger.go:78 in Init) | starting the tagging system
%!s(int=442840760) | INFO | (runner.go:92 in NewRunner) | Runner started with 1 workers.
%!s(int=442840760) | INFO | (collector.go:51 in NewCollector) | Embedding Python 2.7.14 (default, Apr  4 2018, 16:58:02) [GCC 4.7.2]
%!s(int=442840760) | INFO | (file.go:69 in Collect) | File Configuration Provider: searching for configuration files at: /etc/datadog-agent/conf.d
%!s(int=442840760) | INFO | (file.go:69 in Collect) | File Configuration Provider: searching for configuration files at: /opt/datadog-agent/bin/agent/dist/conf.d
%!s(int=442840760) | WARN | (file.go:73 in Collect) | Skipping, open /opt/datadog-agent/bin/agent/dist/conf.d: no such file or directory
%!s(int=442840760) | WARN | (check.go:243 in Configure) | could not get a check instance with the new api: __init__() takes at least 4 arguments (4 given)
%!s(int=442840760) | WARN | (check.go:244 in Configure) | trying to instantiate the check with the old api, passing agentConfig to the constructor
%!s(int=442840760) | WARN | (check.go:269 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (disk).
%!s(int=442845760) | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 25.128.37.38:55411->162.210.111.4:123: i/o timeout
%!s(int=442850760) | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp 25.128.37.38:43615->192.155.90.13:123: i/o timeout
=== Service Checks ===
[
  {
    "check": "ntp.in_sync",
    "host_name": "something**********hj2hx",
    "timestamp": 1523688165,
    "status": 3,
    "message": "",
    "tags": null
  },
  {
    "check": "ntp.in_sync",
    "host_name": "something*************-7b64fd57fb-hj2hx",
    "timestamp": 1523688170,
    "status": 3,
    "message": "",
    "tags": null
  }
]
%!s(int=442850760) | ERROR | (host.go:168 in getCPUInfo) | failed to retrieve cpu info at init time
=========
Collector
=========

  Running Checks
  ==============
    ntp
    ---
      Total Runs: 2
      Metrics: 0, Total Metrics: 0
      Events: 0, Total Events: 0
      Service Checks: 1, Total Service Checks: 2

pbudzon · 2018-04-15T01:41:16Z

@adamgotterer @shine17 check that your firewalls/security groups allow outgoing connections on high ports (or source port 123 or target port 123).
Regular NTP makes connections from port 123 into port 123, but datadog ntp check initiates connection from high ports (example in log above: 25.128.37.38:55411->162.210.111.4:123). I had the same error logged when my firewall was restricting high ports (and only allowing 123<->123).

adamgotterer · 2018-04-16T14:32:10Z

@pbudzon I'm on AWS with DD running on an ECS cluster. That machines running those containers have egress rules for TCP open on ports 0 - 65535. So I don't think its a security group issue.

acmcelwee · 2018-04-16T15:38:36Z

@adamgotterer what about your Network ACL, though?

adamgotterer · 2018-04-16T16:22:44Z

Just double checked the network and ACL and it's allow all traffic on all protocols outbound.

pbudzon · 2018-04-16T16:53:47Z

Remember that network acls are stateless so you need to enable traffic in as well as out.

SleepyBrett · 2018-04-16T17:40:13Z

... why doesn't datadog just trust the host's time?

pbudzon · 2018-04-16T17:48:34Z

This doesn’t have much to do with trust. It’s one of the default checks (like pulling out cpu and memory until) which validates that your system’s time didn’t drift off - you can see the time difference (between system’s time and ntp reported time) in datadog metrics, just like you can see cpu, memory and bunch of other stuff out of the box. If you have ntpd or similar service enabled and working on your server then this check is usually one you can get by without, but still it’s nice to have it. Especially if you’re doing any time-sensitive stuff on the server, like crypto or some authentications.

bompus · 2018-05-15T21:04:58Z

I'm getting the same error. Dedicated server. Ports opened.

2018-05-15 20:55:10 UTC | INFO | (ntp.go:122 in Run) | There was an error querying the ntp host: read udp x.x.x.x:60440->64.113.44.55:123: i/o timeout

bruno-carrier-lookout · 2018-07-10T17:04:06Z

Any update on this?

mad42 · 2018-07-13T05:16:41Z

on a fresh installation :

agent.log INFO UTC | INFO | (ntp.go:123 in Run) | There was an error querying the ntp host: read udp x.x.x.x:52870->138.96.64.10:123: i/o timeout

still broken.

simar7 · 2018-07-17T05:55:43Z

We're experiencing this as well. This seems to be causing the dd agent to stop reporting momentarily and as a result our monitors start to fire for No Data alerts. Any ETA on the fix?

neoghostz · 2018-07-26T06:24:56Z

Could we look at support the ability to pass a runtime var to define an NTP host for the Docker container within AWS so we could leverage their internal endpoint (169.254.169.123)

As a number of organisations aren't always keen to allow NTP in/out of their VPC/Nat Instances.

ChrisMcKee · 2018-08-01T14:13:41Z

Isn't there an option to just turn this off?

asmoran · 2018-08-15T12:02:44Z

We're also having this issue, and it appears to be intermittent. The agent will suddenly recover and start posting metrics briefly, then later fail again for a while.

micahsmith · 2018-09-25T14:55:19Z

It looks as though there are configuration options for NTP (https://docs.datadoghq.com/integrations/ntp/#configuration).

The page reads "The Agent enables the NTP check by default, but if you want to configure the check yourself, edit the file ntp.d/conf.yaml in the conf.d/ folder at the root of your Agent’s configuration directory."

There doesn't seem to be an option to disable NTP checks (although stating that the check is enabled by default does seem to imply there is a way to turn it off), for organizations that want to keep NTP internal, NTP server locations are configurable.

olivielpeau · 2018-10-03T12:24:26Z

Hi all,

Apologies for the late reply. As @micahsmith mentioned, you can configure the Agent's NTP check to query your own/your service provider's NTP servers by following the instructions at https://docs.datadoghq.com/integrations/ntp/#configuration.

Also, as with any other Agent check that's enabled by default, you can disable the check completely by removing the file at <agent_conf_dir>/conf.d/ntp.d/conf.yaml.default and restarting the Agent (on a standard Linux install, <agent_conf_dir> is /etc/datadog-agent).

That said, we don't recommend disabling the NTP check entirely as:

it allows you to know when your host's clock is skewed
an Agent running on a host on which the system clock is significantly skewed (approx. >60s) may report data so much in the past or future that it'll affect your graphs and monitors on Datadog.

We've implemented significant improvements to the NTP check in v6.4.0 and 6.5.0, if you're having issues with the NTP check please upgrade to Agent >= 6.5.0. If you still run into issues with the NTP check after upgrading please reach out to our support team and send them your Agent's logs.

ahharu · 2019-04-04T11:37:12Z

Having this issue on K8S Deployment on AWS , version of agent 6.10.2

dabcoder · 2019-05-06T17:22:58Z

@ahharu Thanks for the heads up, could you send us a note via [email protected] with a flare and some details about your configuration? We could then assess the situation.

albertvaka · 2019-08-20T14:38:28Z

We have merged a couple PRs for the upcoming 6.14 release that should make this better. Please let us know if this is still an issue for you after upgrading.

olivielpeau added the [deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. label Oct 3, 2018

This was referenced Aug 19, 2019

[NTP] Use the timeout from the config #4035

Merged

[NTP] Lower log level for query errors, warn if recurrent #4037

Merged

TPS mentioned this issue Oct 26, 2024

DataDog NTP AdguardTeam/AdguardFilters#191645

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

There was an error querying the ntp host #1532

There was an error querying the ntp host #1532

adamgotterer commented Mar 27, 2018 •

edited

Loading

shine17 commented Apr 14, 2018 •

edited

Loading

shine17 commented Apr 14, 2018

pbudzon commented Apr 15, 2018

adamgotterer commented Apr 16, 2018

acmcelwee commented Apr 16, 2018

adamgotterer commented Apr 16, 2018

pbudzon commented Apr 16, 2018 via email •

edited

Loading

SleepyBrett commented Apr 16, 2018

pbudzon commented Apr 16, 2018 via email •

edited

Loading

bompus commented May 15, 2018

bruno-carrier-lookout commented Jul 10, 2018

mad42 commented Jul 13, 2018

simar7 commented Jul 17, 2018

neoghostz commented Jul 26, 2018

ChrisMcKee commented Aug 1, 2018

asmoran commented Aug 15, 2018 •

edited

Loading

micahsmith commented Sep 25, 2018 •

edited

Loading

olivielpeau commented Oct 3, 2018

ahharu commented Apr 4, 2019

dabcoder commented May 6, 2019

albertvaka commented Aug 20, 2019

There was an error querying the ntp host #1532

There was an error querying the ntp host #1532

Comments

adamgotterer commented Mar 27, 2018 • edited Loading

shine17 commented Apr 14, 2018 • edited Loading

shine17 commented Apr 14, 2018

pbudzon commented Apr 15, 2018

adamgotterer commented Apr 16, 2018

acmcelwee commented Apr 16, 2018

adamgotterer commented Apr 16, 2018

pbudzon commented Apr 16, 2018 via email • edited Loading

SleepyBrett commented Apr 16, 2018

pbudzon commented Apr 16, 2018 via email • edited Loading

bompus commented May 15, 2018

bruno-carrier-lookout commented Jul 10, 2018

mad42 commented Jul 13, 2018

simar7 commented Jul 17, 2018

neoghostz commented Jul 26, 2018

ChrisMcKee commented Aug 1, 2018

asmoran commented Aug 15, 2018 • edited Loading

micahsmith commented Sep 25, 2018 • edited Loading

olivielpeau commented Oct 3, 2018

ahharu commented Apr 4, 2019

dabcoder commented May 6, 2019

albertvaka commented Aug 20, 2019

adamgotterer commented Mar 27, 2018 •

edited

Loading

shine17 commented Apr 14, 2018 •

edited

Loading

pbudzon commented Apr 16, 2018 via email •

edited

Loading

pbudzon commented Apr 16, 2018 via email •

edited

Loading

asmoran commented Aug 15, 2018 •

edited

Loading

micahsmith commented Sep 25, 2018 •

edited

Loading