[BUG] failed to send, dropping xxx traces to intake at xxx after 3 retries ([Errno 111] Connection refused)

[sachams]

Agent Environment

• Agent version 3.30.8
• Python ddtrace version 1.12.6

	"sourceLocation": {
		"file": "/usr/local/lib/python3.10/site-packages/ddtrace/internal/writer.py",
		"line": 464,
		"function": "flush_queue"
	}

Describe what happened:

We frequently see this error in all our environments, for example:

failed to send, dropping 23 traces to intake at http://10.154.0.10:8126/v0.5/traces after 3 retries ([Errno 111] Connection refused)

DataDog is otherwise functioning fine. Would like to stop these errors being generated though.

Describe what you expected:

No errors.

Steps to reproduce the issue:

• Deploy datadog helm 3.30.8
• Use ddtrace in a Python application

Additional environment details (Operating System, Cloud provider, etc):

• Running on python:3.10.10-slim-bullseye on GKE

[darkwind-noveo]

U should see happening datadog agent logs at this time.
Highly likely your agent is rapidly restarting, because each time it exits for some reason

[ricardodani]

I'm having the same, would like to stop it too

[vkaracic]

Same issue here, @sachams have you found a fix for this?

[luciidlou]

bump

[miguelgf]

Do we have any ETA for this to be solved or a workaround?

[jmborsani]

+1

[domino14]

please fix

[victorct-tradeengage]

Same issue, here but this is the log we have: failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces after 3 retries. Happens on AWS ECS Fargate with Datadog as a sidecar and only on startup so far.

[suemayaheldursi]

anyone know why this is? traces seem to be coming through ok but I keep getting this:

failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces

Running on python with ecs fargate and datadog agent as sidecar.

Docker image: python:3.10.10-slim
ddtrace: ^2.12.2

[UtpalCerta]

Same issue, here we have log as : failed to send, dropping 2 traces to intake at http://10.0.169.74:8126/v0.5/traces after 3 retries

Application running on Python with ECS backed by EC2 instance & datadog agent as daemon agent

Versions
Datadog Image : 7.41.1
Python : 3.9.18
DDtrace : 2.9.2

[yixuanshao1991]

This error disappeared after holding python app startup for 10s to wait DD agent.
Tested in AWS ECS sidecar DD agent.
DDtrace: 2.11.2
Python: 3.12

[davidwinter]

What is the status of this issue? Is there no workaround other than to delay the start of the application launching? Currently this error is being caught and captured by our Sentry integration which is not ideal at all.

[pzelnip]

Am getting this as well when running my application's tests with Pytest. ddtrace version 2.18.1 and sentry-sdk 2.20.0. Workaround for me was to set the DD_TRACE_ENABLED environment variable to false for my tests.

[johnecon]

Also getting this running my app in AWS Lambda

[the-real-mathew-moon]

Also hitting this when running in AWS Lambda.