retry to checkin are untimely

[Jimyhjy]

when checkin failed，why need long time for retry next times?

this unbuffered channel，Is it possible for it to cause a blocking wait?

[cmacknz]

The screenshot is cutting off what the retry should be, in the retry_after_ns field. Can you include the entire log line?

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 199 to 201 in de39cae

	f.log.Warnw("Possible transient error during checkin with fleet-server, retrying",
	"error.message", err, "request_duration_ns", took, "failed_checkins", f.checkinFailCounter,
	"retry_after_ns", bo.NextWait())

I can see there was only 1 failure so the delay here seems unusually high, but this will let us confirm if the retry was calculated wrong or if we were stuck somewhere we shouldn't be as you suggest.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[Jimyhjy]

this is the complete log,"retry_after_ns":647791371979, about 10 minute, why so long？

{"log.level":"warn","@timestamp":"2025-07-04T07:08:12.938Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":194},"message":"Possible transient error during checkin with fleet-server, retrying","log":{"source":"elastic-agent"},"error":{"message":"could not decode the response, raw response: \r\n<title>504 Gateway Time-out</title>\r\n\r\n

504 Gateway Time-out

\r\n

---

nginx\r\n\r\n\r\n"},"request_duration_ns":300001132274,"failed_checkins":1,"retry_after_ns":647791371979,"ecs.version":"1.6.0"}

Is it affected by this configuration？

@cmacknz

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Thank you we shouldn't be using the max 10m backoff on the first retry, something is wrong here.

We reset our backoff with bo.Reset() before entering the retry loop which is supposed to prevent this:

elastic-agent/internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

Lines 186 to 208 in 5934e75

	func (f *FleetGateway) doExecute(ctx context.Context, bo backoff.Backoff) (*fleetapi.CheckinResponse, error) {
	bo.Reset()
	// Guard if the context is stopped by a out of bound call,
	// this mean we are rebooting to change the log level or the system is shutting us down.
	for ctx.Err() == nil {
	f.log.Debugf("Checking started")
	resp, took, err := f.execute(ctx)
	if err != nil {
	f.checkinFailCounter++
	// Report the first two failures at warn level as they may be recoverable with retries.
	if f.checkinFailCounter <= 2 {
	f.log.Warnw("Possible transient error during checkin with fleet-server, retrying",
	"error.message", err, "request_duration_ns", took, "failed_checkins", f.checkinFailCounter,
	"retry_after_ns", bo.NextWait())
	} else {
	f.log.Errorw("Cannot checkin in with fleet-server, retrying",
	"error.message", err, "request_duration_ns", took, "failed_checkins", f.checkinFailCounter,
	"retry_after_ns", bo.NextWait())
	}
	if !bo.Wait() {

Our backoff looks like it has two pieces of state, duration and nextRand but it looks like the problem is likely that our Reset method doesn't reset nextRand it potentially can be at the max if there were previous retries in the past.

elastic-agent/internal/pkg/core/backoff/equal_jitter.go

Lines 50 to 58 in 5934e75

	func (b *EqualJitterBackoff) Wait() bool {
	backoff := b.NextWait()
	// increase duration for next wait.
	b.nextRand = rand.N(b.duration)
	b.duration *= 2
	if b.duration > b.max {
	b.duration = b.max
	}

elastic-agent/internal/pkg/core/backoff/equal_jitter.go

Lines 38 to 41 in 5934e75

	func (b *EqualJitterBackoff) Reset() {
	// Allow to sleep at least the init period on the first wait.
	b.duration = b.init * 2
	}

[Jimyhjy]

When will this bug be fixed？

[cmacknz]

We can't promise specific release dates but it is scheduled to be worked on soon.

[Jimyhjy]

Ok, thank you!