Skip to content

Commit 55e14df

Browse files
committed
fix(logserver): Prevent race condition during reconciliation
Resolves an issue where the logserver StatefulSet would hang for over 10 minutes during a rollout, a problem unique to this component. Pod events showed repeated "FailedMount" warnings, eventually timing out with the error: "Unable to attach or mount volumes: timed out waiting for the condition". The root cause was a race condition between the operator's reconciliation loop and the Kubernetes controller managing the StatefulSet update. Immediately after triggering a rollout, the operator would proceed to reconcile the associated PVC. This interfered with the kubelet's process of detaching the volume from the old pod and attaching it to the new one, causing the prolonged timeout. This commit fixes the race condition by ensuring that the DeployLogserver function exits its reconciliation loop immediately after a StatefulSet update has been triggered. This gives the Kubernetes volume controller uninterrupted time to manage the PVC handover. This change aligns the logserver controller's behavior with all other StatefulSet controllers in the operator, which already followed this pattern, correcting a historical inconsistency that only affected logserver. Change-Id: I164ef03e0e4ef8557a1ec5effb0415b77a7c053f
1 parent 27f840b commit 55e14df

File tree

2 files changed

+4
-0
lines changed

2 files changed

+4
-0
lines changed

controllers/logserver.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,9 @@ func (r *SFController) DeployLogserver() bool {
303303
sts.Spec.Template.Spec.HostAliases = base.CreateHostAliases(r.cr.Spec.HostAliases)
304304

305305
current, stsUpdated := r.ensureStatefulset(storage.StorageClassName, sts)
306+
if stsUpdated {
307+
return false
308+
}
306309

307310
pvcReadiness := r.reconcileExpandPVC(logserverIdent+"-"+logserverIdent+"-0", r.cr.Spec.Logserver.Storage)
308311

doc/reference/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ All notable changes to this project will be documented in this file.
1919
- Fix a few issues with the nodeAffinity setting logic, where the node affinity of a statefulset
2020
would only be updated when its annotations are changed; and where it would hang if the statefulset's
2121
replicas are set to 0.
22+
- Fix a race condition in the logserver controller that could cause PVCs to get stuck during operator upgrades or resource updates.
2223

2324
## [v0.0.58] - 2025-09-05
2425

0 commit comments

Comments
 (0)