The check command used by the service runtime does not guarantee that the service is running

[kaanyalti]

When the agent is starting a service runtime, it uses a check command to validate that the relevant external services are installed. This function just calls the prescribed check command in the service spec file. In the context of the endpoint service, this command does not seem to account for whether the endpoint is running or not. If endpoint service is not running, the check command can still return without any errors.

For confirmed bugs, please report:

• Version: Main branch
• Operating System: Ubuntu 24.04.2 LTS and CentOS Stream 9
  • Mac is most likely affected as well
  • Windows is most likely not affected

Steps to Reproduce:
I ran into this while working on the following #6394, so the steps will include using tamper protected agent-endpoint; however, the bug should be reproducible without tamper protection as well.

• Create an es deployment
• In fleet ui create a policy, add endpoint integration, and enable tamper protection
• Add an agent following deb/rpm instructions
• Validate that both the agent and endpoint are running and are healthy (sudo elastic-agent status --output full)
• Stop the endpoint service systemctl stop ElasticEndpoint
• Remove endpoint's vault directory at /opt/Elastic/Endpoint/state/vault
• Install the same version of the agent with dpkg -i elastic-agent-<SAME VERSION>.deb
• After waiting a bit, validate that the endpoint service is still not running,
  and that the agent is unhealthy.

[kaanyalti]

@cmacknz @ebeahan
Creating this as a result of the discussion in the team slack channel regarding the tamper protected deb/rpm upgrades.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

I am going to ask the defend team if this should be fixed in the check command or in agent, probably something should be done in the endpoint-security command unless we want to call install again. We don't know what the service name is technically since it isn't in the spec file, though we could just assume it isn't going to change.

[ebeahan]

@kaanyalti this only impacts endpoint?

[cmacknz]

Summarizing a separate conversation with @bjmcnic:

• The current behavior is fine and not urgent to fix. The reason is that endpoint will eventually show as degraded or failed when manually stopped as it can't checkin with agent. This gives the user a clear signal that something is wrong on the agent host, but not exactly what that is.
• Endpoint could have the check command detect that endpoint is installed but not running, and in that case we could improve the error to say that endpoint was stopped on the host.
  • We could and/or should have agent run systemctl status to determine if endpoint is stopped and improve the agent error messages ourselves.
  • We could add a configuration flag for whether agent should auto-start endpoint in this case. We likely don't want this by default as it will hide situations where someone is messing with the host or has crashed which are rare and we'd want to know about them.
  • We could perhaps instead only auto-start endpoint after it has failed to checkin in for a period and we have notified the user there is a problem.
    • It would be interesting if we could auto-collect diagnostics as part of this.
  • This situation can be fixed remotely already by moving the agent to a policy without Defend then moving it back, as we'll re-install Defend in that case.