What happened:
While reviewing the metrics collection logic in the autoscaler, I found that the Prometheus metrics decoding loop in pkg/controller/autoscaler/metric_collector.go only exits on io.EOF. However, non-EOF errors (e.g., network timeouts, malformed responses, connection resets) cause the loop to spin indefinitely with continue, leading to:
- Goroutine leaks — the collector goroutine never terminates
- CPU spin — tight loop on persistent decode errors
- Autoscaler crash — under sustained load with flaky metrics endpoints
This pattern was partially addressed in related reliability work (see PR #952), but the metric collector's decode loop lacks proper error handling and backoff.
The problematic code:
for {
var metric prom2json.Family
if err := decoder.Decode(&metric); err != nil {
if err == io.EOF {
break
}
// BUG: continues looping on non-EOF errors without backoff
continue
}
}
What happened:
While reviewing the metrics collection logic in the autoscaler, I found that the Prometheus metrics decoding loop in
pkg/controller/autoscaler/metric_collector.goonly exits onio.EOF. However, non-EOF errors (e.g., network timeouts, malformed responses, connection resets) cause the loop to spin indefinitely withcontinue, leading to:This pattern was partially addressed in related reliability work (see PR #952), but the metric collector's decode loop lacks proper error handling and backoff.
The problematic code: