Retry graceful shutdown on vSphere VM deletion until timeout

[Copilot]

During vSphere VM deletion, if vm.ShutdownGuest() fails on first attempt (VMware Tools not ready, transient API failure), the entire deletion fails. After 3 retries, Rancher abandons the deletion leaving orphaned VMs.

Changes

Unified retry loop with precise timing:

• Single loop using time.Now() and deadline-based timing for accuracy
• Continuously retries d.Stop() until GracefulShutdownTimeout is reached
• Checks VM state on every iteration, allowing early exit even if shutdown initiation fails
• Log each failed attempt at Debug level with error details

Timeout tracking:

• Uses time.Since() to track elapsed time from start
• Ensures total time never exceeds GracefulShutdownTimeout
• Smart sleep logic that prevents exceeding the deadline

Fallback behavior:

• Proceeds to d.Kill() (hard power off) if graceful shutdown can't be initiated within timeout
• Also falls back if graceful shutdown doesn't complete within timeout

Before:

if err := d.Stop(); err != nil {
    return err  // Fails immediately
}
// wait for machine to stop...

After:

start := time.Now()
deadline := start.Add(time.Duration(d.GracefulShutdownTimeout) * time.Second)

for time.Now().Before(deadline) {
    if !gracefulShutdownInitiated {
        if err := d.Stop(); err != nil {
            // Log and retry
        } else {
            gracefulShutdownInitiated = true
        }
    }
    
    // Always check state for early exit
    if machineState == state.Stopped {
        break
    }
    
    // Sleep without exceeding deadline
    time.Sleep(...)
}
// If not stopped, proceed to d.Kill()

[!WARNING]

Original prompt

There is an issue with vSphere nodes deletion if GracefulShutdownTimeout is set but rancher/machine is failing to actually kick off graceful shutdown and then machine deletion fails. After 3 retries Rancher fails to delete machine and users

We want to update the logic to where we continue attempting to kick off graceful shutdown until 'GracefulShutdownTimeout' is met (and if we succeed to kick off graceful shutdown before 'GracefulShutdownTimeout' is met we wait for graceful shutdown to complete until total wait time between waiting to kick off and waiting for actual shutdown exceeds GracefulShutdownTimeout'). Worth nothing that if we fail to kick off graceful shutdown within GracefulShutdownTimeout we should move on to hard restart (d.Kill`).

Note graceful shutdown was implemented as part of #222.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[jferrazbr]

@copilot please update the description based on the latest changes pushed to the branch

[Copilot]

@copilot please update the description based on the latest changes pushed to the branch

Updated the PR description to reflect the refactored implementation with improved timing using time.Now() and a single unified loop (commit 680b751).

[jiaqiluo]

LGTM