Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Agent conflicts with Docker Registry on port 5000 #4229

Open
alexeyzimarev opened this issue Sep 29, 2019 · 13 comments
Open

Agent conflicts with Docker Registry on port 5000 #4229

alexeyzimarev opened this issue Sep 29, 2019 · 13 comments
Labels
[deprecated] team/agent-shared-components Deprecated. Use team/agent-configuration or team/agent-runtimes labels instead.

Comments

@alexeyzimarev
Copy link

Describe what happened:

After installing the agent on our GitLab server, we encountered issues with GitLab Container Registry. The issue didn't appear immediately after the agent was installed but after the scheduled server update follower by the restart.

Suddenly, all attempts to push to the container registry started to fail with 404 error. It significantly disturbed the work of our development team, because it was working and out of the blue stopped working.

After checking GitLab omnibus logs, I found out that the Docker registry keeps restarting, reporting that 127.0.0.1:5000 is already in use. Then I found out that the agent is listening on that port.

I had to change the agent settings so it doesn't use port 5000 and I don't really know what consequences it would have on the agent itself.

Describe what you expected:

Since GitLab integration is one of the standard integrations of Datadog, I was not expecting that it will come in direct conflict of the widely used GitLab component, such as the container registry.

Steps to reproduce the issue:

Install GitLab using Omnibus and enable the container registry. Check if everything works as expected. Install the Datadog agent on the same machine, reboot the machine and observe that the GHitLab container registry keeps restarting because port 5000 is already in use.

Additional environment details (Operating System, Cloud provider, etc):

Ubuntu 18.04

@OneCricketeer
Copy link

OneCricketeer commented Nov 19, 2019

That is the go-expvar server port. It can be changed without conflict

https://docs.datadoghq.com/integrations/go_expvar/

@olivielpeau
Copy link
Member

Indeed, the port (default: 5000) on which the Agent listens locally can be changed with the expvar_port option of the Agent config (see

## @param expvar_port - integer - optional - default: 5000
## The port for the go_expvar server.
#
# expvar_port: 5000
). If it's set to a different value than its default, the Agent will continue to work, all the parts of the Agent that use this port will pick up change.

The only exception is if you've set up the go_expvar Agent integration to monitor the Agent itself (on localhost:5000). If you change the expvar port of the Agent, and want the go_expvar check to continue monitoring the Agent, you'll have to update the port it connects to accordingly, as documented in https://docs.datadoghq.com/integrations/go_expvar/ (and mentioned by @Cricket007 ).

We are aware that there is a chance that port 5000 is used by other processes on some hosts, so we're considering changing the Agent's expvar port to a different value (less likely to conflict with other processes) in a future version of the Agent. We'll keep this card updated when that happens.

@olivielpeau olivielpeau added the [deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. label Nov 19, 2019
@colinmollenhour
Copy link

Port 5000 also conflicts with GitLab omnibus container.

@pablocompagni-contractorvp

I'm running into this issue with an ECS Fargate container, does anyone know which env var is able to change the DD agent port from 5000 to something else? thanks

@sgnn7
Copy link
Contributor

sgnn7 commented Apr 30, 2021

@pablocompagni-contractorvp You can see it in one of the earlier posts but the setting is expvar_port in datadog.yaml. Alternatively, you may be able to use DD_EXPVAR_PORT env var to set this (though I did not test this).

@archy-bold
Copy link

Can confirm setting DD_EXPVAR_PORT works for me.

@MGough
Copy link

MGough commented Nov 11, 2021

I encountered this issue and it took a while to work out what was going on. I was following this documentation:
https://docs.datadoghq.com/integrations/ecs_fargate/?tab=fluentbitandfirelens

The issues were:

  • The documentation doesn't state that the datadog agent uses port 5000
  • The configuration provided makes all containers essential so it kept shutting down making debugging trickier (hard to tell which container exited first)
  • The configuration provided means that the datadog agent doesn't log to AWS so I couldn't see any logs showing the datadog container error

Mentioning port 5000 in those docs would've saved me a lot of pain!

Setting DD_EXPVAR_PORT worked for me too, although my first attempt 5001 proved to be already in use.

@boomshadow
Copy link

This was quite the unexpected thing to wake up to this morning. We did a GitLab scheduled maintenance reboot overnight, and then woke up to all our CI pipelines failing. It was a mad scramble for a few hours to figure out that our DD agent was port binding before the Docker registry :(

Thanks for the information here! I was able to get this resolved with your help. I do hope that Datadog considers changing the default port to something far less common.

A handy trick is to set the expvar_port to 0 (zero). The kernel will give it a random, guaranteed-to-be unassigned port.

@htims1989
Copy link

Just run into this following: https://www.datadoghq.com/blog/deploy-dotnet-core-aws-fargate/

Especially awkward since ASP.NET Core containers default to port 5000!

@willogden
Copy link

Also ran into this. For those also needing port 5001, this is used by the agent CLI service and can be changed using the DD_CMD_PORT environment variable.

@baarney
Copy link

baarney commented Nov 6, 2023

Also conflicts with default nginx config on AWS Elastic Beanstalks.

By default, Elastic Beanstalk configures the proxy to forward requests to your application on port 5000.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/nodejs-platform-proxy.html

@sgnn7 sgnn7 added [deprecated] team/agent-shared-components Deprecated. Use team/agent-configuration or team/agent-runtimes labels instead. and removed [deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. labels Nov 8, 2023
@sgnn7
Copy link
Contributor

sgnn7 commented Nov 8, 2023

Hi @baarney (and others in the thread),
As mentioned in our older comment on it, we are aware that there is a chance that port 5000 is used by other processes on some hosts and we are still considering changing the Agent's default expvar port to a different value in a future version of the Agent. For context some of the bigger concerns here around changing that value are around ensuring that users of expvar/go_expvar functionality do not get unexpected changes that break their currently-working workflows.

@rasatlas
Copy link

On ubuntu 20.04 got to /etc/datadog-agent/datadog.yaml file and uncomment the line containing expvar_port and change its value to a port you don't use.

expvar_port: 6062

Before everything stopping datadog-agent service and restarting it after the value change is needed.

This has solved my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[deprecated] team/agent-shared-components Deprecated. Use team/agent-configuration or team/agent-runtimes labels instead.
Projects
None yet
Development

No branches or pull requests