[8.19] (backport #8737) Move beat receiver component logic to the otel manager

[mergify]

NOTE: This is an alternative implementation of #8529, which adds the new functionality to the existing Otel manager, instead of creating a new object for it.

What does this PR do?

Move beat receiver component logic to the otel manager.

This logic consists of two tasks conceptually:

• Translating agent configurations into beats receiver configurations for the otel collector.
• Translating otel collector statuses into component states.

Up until now, the logic for these was haphazardly spread across the agent coordinator. This PR moves all of it into the OtelManager, which can now run both raw otel collector configurations and agent components in a single otel collector instance.

The OtelManager now encapsulates all the logic involved in interfacing between the agent coordinator and the otel collector. In the near future, it will also take on additional responsibilities, like generating diagnostics for components it runs.

The only new logic this PR introduces lives in the new manager's main loop, and has to do with how updates and configurations are moved around. The rest is either existing logic moved to a new location, and new tests for that old logic.

Why is it important?

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Related issues

• Relates Ensure that diagnostics contain all information necessary to debug beat receivers when using the OTel runtime #8208

---

This is an automatic backport of pull request #8737 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 503421f has failed:

On branch mergify/bp/8.19/pr-8737
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 503421fc8.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .gitignore
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_test.go
	modified:   internal/pkg/otel/manager/common.go
	modified:   internal/pkg/otel/manager/execution_subprocess.go
	modified:   internal/pkg/otel/manager/manager.go
	modified:   internal/pkg/otel/manager/manager_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/agent/application/coordinator/coordinator.go
	both modified:   internal/pkg/agent/application/coordinator/coordinator_unit_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 86c9bb8

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• :junit: Junit annotate
• Merge coverage reports
• :sonarqube: Continuous Code Inspection
• Packaging: linux/amd64 rpm
• Packaging: linux/amd64 rpm
• Packaging: Ubuntu x86_64 FIPS
• Packaging: Ubuntu x86_64 FIPS
• Packaging: linux/arm64 tar.gz
• Packaging: linux/arm64 tar.gz
• Packaging: Ubuntu arm64 FIPS
• Packaging: Ubuntu arm64 FIPS
• Packaging: windows/amd64 zip
• Packaging: windows/amd64 zip
• Packaging: Containers linux/amd64
• Packaging: Containers linux/arm64
• Packaging: Containers linux/amd64 FIPS
• Packaging: Containers linux/arm64 FIPS

History

cc @swiatekm

[elastic-sonarqube]

Quality Gate passed

Issues
4 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
81.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube