Fix data race in checkin API and improve overall performance

[blakerouse]

What is the problem this PR solves?

Fix data race in checkin API to prevent multiple Fleet Servers handling the same Elastic Agent from overwriting each other. Ensures that the Elastic Agent document is updated before a response back from check-in, to ensure that the state between Fleet and Elastic Agent is always consistent.

Improves the checkin logic to make it easier to understand and removes each moving parts. Removes the need for an extra ticker in each long poll connection select. Removes the need to compute unique Elastic Agent bodies to send in the MUpdate. This reduces the amount of memory allocations and the amount of data that is sent over the MUpdate.

How does this PR solve the problem?

It solves the issue by ensuring to write the Elastic Agent document before sending a response, which ensures that if the Elastic Agent quickly makes another connection to a different Fleet Server that it is working with the previous state that was written by the previous checkin.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• [ ] I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (no changes in this PR require this)

Checklist

• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation (none related in this PR, all internal)
• [ ] I have made corresponding change to the default configuration files (none)
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes CheckIn bulker 10 second window is a problem #5793
• Closes Improve checkin bulker performance #5563

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Licenses	0	0	0	0	0 issues
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[pchila]

Only went through handleCheckin* and I left a couple of comments. May come back with more once I go over the rest of the PR.

I also think that @michel-laterman could have a look at this since he should have a better context around agent checkin in fleet-server

[michel-laterman]

Overall, I really like the refactor.
My biggest concern is the possbile 2nd ES call to remove the audit/unenrolled attributes

[blakerouse]

@pchila @michel-laterman Can you give this another review?

[pchila]

Thank you for addressing my comments. LGTM