Agents remains in UPG_SCHEDULED state when upgrade action is cancelled

[juliaElastic]

Bug discovered in scale tests: https://github.com/elastic/ingest-dev/issues/5774

There is a testcase that cancels upgrade action, which leaves agents in UPG_SCHEDULED state.
This causes the next testcase auto upgrade agents fail as auto upgrade doesn't pick up agents with upgrade details in non-terminal state.

This can be reproduced by starting an upgrade action with a rollout duration (e.g. 10m) and then cancelling the action.
The agents are stuck in scheduled state.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[amolnater-qasource]

Hi @juliaElastic
Just an update that we had a similar issue reported here #4293 where agent remains stuck on cancelling the schedule upgrade.

Could you please confirm if these are related?
Thanks!!

[juliaElastic]

@amolnater-qasource Thanks for the link, it seems to be the same issue. cc @cmacknz @ebeahan

[cmacknz]

#4293 (comment) was my original assessment of where this is going wrong. Closing that issue as a duplicate and using this one.

[mbudge]

We've also got a few agents impacted by this.

We'll stop using the scheduled upgrade feature until this is fixed.

Is this highlighted in the release notes?

Thanks!

[cmacknz]

Is this highlighted in the release notes?

Following up on the release notes, confirming the affected versions and working to find a work around. We have the internal support case for this as well now.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

The work around is to force the agent to upgrade via the /fleet/agents API:

curl --request POST \
  --url https://<KIBANA_HOST>/api/fleet/agents/<AGENT_ID>/upgrade \
  --user "<SUPERUSER_NAME>:<SUPERUSER_PASSWORD>" \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: as' \
  --data '{"version": "<VERSION>","force": true}'