Move beat receiver component logic to the otel manager

[swiatekm]

NOTE: This is an alternative implementation of #8529, which adds the new functionality to the existing Otel manager, instead of creating a new object for it.

What does this PR do?

Move beat receiver component logic to the otel manager.

This logic consists of two tasks conceptually:

• Translating agent configurations into beats receiver configurations for the otel collector.
• Translating otel collector statuses into component states.

Up until now, the logic for these was haphazardly spread across the agent coordinator. This PR moves all of it into the OtelManager, which can now run both raw otel collector configurations and agent components in a single otel collector instance.

The OtelManager now encapsulates all the logic involved in interfacing between the agent coordinator and the otel collector. In the near future, it will also take on additional responsibilities, like generating diagnostics for components it runs.

The only new logic this PR introduces lives in the new manager's main loop, and has to do with how updates and configurations are moved around. The rest is either existing logic moved to a new location, and new tests for that old logic.

Why is it important?

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Related issues

• Relates Ensure that diagnostics contain all information necessary to debug beat receivers when using the OTel runtime #8208

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b chore/otel-manager-components upstream/chore/otel-manager-components
git merge upstream/main
git push upstream chore/otel-manager-components

[pkoutsovasilis]

I did left some comments, @swiatekm take a look and tell me what you think. PS: I am more than happy to have a dedicated sync about that 🙂

[pkoutsovasilis]

thanks for addressing my comments @swiatekm. As you have already mentioned, there are points that we should revisit in the future but this PR feels to me a step towards the right direction:

• the final otel config is produced inside the otel manager
• statuses are fabricated and emitted from the manager. A future improvement here is to investigate if we could emit only one status and satisfy both the collector and components status needs, which probably will help in simplifying the code
• sure the manager loop got a little bit "heavier" but we can try in the future to improve code readability

So code changes wise this counts as an improvement. Now that said, I think we have an issue 😄

I did compile and run this on my machine and for both execution modes I see only the following reported through elastic-agent status --output full

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: c3189bbf-0e33-4b6b-aab7-80a4a3a0647b
   │  ├─ version: 9.2.0
   │  └─ commit: 86fc4fcea1209c63a3b810f941449b743bc0b440
   └─ system/metrics-default
      ├─ status: (STARTING) Starting: spawned pid '1284'
      ├─ system/metrics-default
      │  ├─ status: (STARTING) Starting: spawned pid '1284'
      │  └─ type: OUTPUT
      └─ system/metrics-default-unique-system-metrics-input
         ├─ status: (STARTING) Starting: spawned pid '1284'
         └─ type: INPUT

which makes me believe that there is a tiny bit somewhere in the status channels that is missed!? but is this reproducible also on your end?

[elastic-sonarqube]

Quality Gate passed

Issues
4 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
81.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: f0305ec

Failed CI Steps

• fips:x86_64:sudo-true:fleet
• fips:arm64:sudo-true:fleet

History

• 💔 Build #23989 failed 86fc4fc
• 💚 Build #23970 succeeded f76030a
• 💔 Build #23635 failed 176ece7
• 💔 Build #23280 failed 6ef024f
• 💔 Build #23278 failed 4302811
• 💔 Build #23271 failed 54df9df

cc @swiatekm

[swiatekm]

thanks for addressing my comments @swiatekm. As you have already mentioned, there are points that we should revisit in the future but this PR feels to me a step towards the right direction:

* the final otel config is produced inside the otel manager

* statuses are fabricated and emitted from the manager. A future improvement here is to investigate if we could emit only one status and satisfy both the collector and components status needs, which probably will help in simplifying the code

* sure the manager loop got a little bit "heavier" but we can try in the future to improve code readability

So code changes wise this counts as an improvement. Now that said, I think we have an issue 😄

I did compile and run this on my machine and for both execution modes I see only the following reported through elastic-agent status --output full

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: c3189bbf-0e33-4b6b-aab7-80a4a3a0647b
   │  ├─ version: 9.2.0
   │  └─ commit: 86fc4fcea1209c63a3b810f941449b743bc0b440
   └─ system/metrics-default
      ├─ status: (STARTING) Starting: spawned pid '1284'
      ├─ system/metrics-default
      │  ├─ status: (STARTING) Starting: spawned pid '1284'
      │  └─ type: OUTPUT
      └─ system/metrics-default-unique-system-metrics-input
         ├─ status: (STARTING) Starting: spawned pid '1284'
         └─ type: INPUT

which makes me believe that there is a tiny bit somewhere in the status channels that is missed!? but is this reproducible also on your end?

Can you post your exact configuration? I tested

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: "elastic"
    password: "..."

agent:
  logging:
    to_stderr: true
  monitoring:
    enabled: false

inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  - data_stream:
      dataset: system.memory
    metricsets:
    - memory
  - data_stream:
      dataset: system.network
    metricsets:
    - network
  - data_stream:
      dataset: system.filesystem
    metricsets:
    - filesystem
  type: system/metrics
  use_output: default
  _runtime_experimental: otel

and got:

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: fd4b48da-1a8a-42f3-aca2-321df32b45d7
   │  ├─ version: 9.2.0
   │  └─ commit: 86fc4fcea1209c63a3b810f941449b743bc0b440
   └─ system/metrics-default
      ├─ status: (HEALTHY) HEALTHY
      ├─ system/metrics-default
      │  ├─ status: (HEALTHY) Healthy
      │  └─ type: OUTPUT
      └─ system/metrics-default-unique-system-metrics-input
         ├─ status: (HEALTHY) Healthy
         └─ type: INPUT

and if I switch back to process mode for the system/metrics input:

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: fd4b48da-1a8a-42f3-aca2-321df32b45d7
   │  ├─ version: 9.2.0
   │  └─ commit: 86fc4fcea1209c63a3b810f941449b743bc0b440
   └─ system/metrics-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '517716'
      ├─ system/metrics-default
      │  ├─ status: (HEALTHY) Healthy
      │  └─ type: OUTPUT
      └─ system/metrics-default-unique-system-metrics-input
         ├─ status: (HEALTHY) Healthy
         └─ type: INPUT

[pkoutsovasilis]

@swiatekm 👋 this is my config as reported from the inspect sub-command

agent:
  logging:
    to_stderr: true
  monitoring:
    _runtime_experimental: otel
    enabled: true
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  - data_stream:
      dataset: system.memory
    metricsets:
    - memory
  - data_stream:
      dataset: system.network
    metricsets:
    - network
  - data_stream:
      dataset: system.filesystem
    metricsets:
    - filesystem
  type: system/metrics
  use_output: default
outputs:
  default:
    api_key: <REDACTED>
    hosts:
    - 127.0.0.1:9200
    preset: balanced
    type: elasticsearch

for the subprocess approach I would expect to see something like this (PS: I get the same for the in-process approach without the extensions section)

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ info
   │  ├─ id: f24c5572-89c7-450e-87b8-14cfd8741a83
   │  ├─ version: 9.2.0
   │  └─ commit: 86fc4fcea1209c63a3b810f941449b743bc0b440
   ├─ beat/metrics-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  ├─ beat/metrics-monitoring
   │  │  ├─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   │  │  └─ type: OUTPUT
   │  └─ beat/metrics-monitoring-metrics-monitoring-beats
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ filestream-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  ├─ filestream-monitoring
   │  │  ├─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   │  │  └─ type: OUTPUT
   │  └─ filestream-monitoring-filestream-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ http/metrics-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  ├─ http/metrics-monitoring
   │  │  ├─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   │  │  └─ type: OUTPUT
   │  └─ http/metrics-monitoring-metrics-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ system/metrics-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '67276'
   │  ├─ system/metrics-default
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ system/metrics-default-unique-system-metrics-input
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   └─ extensions
      ├─ status: StatusOK
      ├─ extension:healthcheckv2/a8aa1840-f3cd-4464-9b0b-68251f8ce626
      │  └─ status: StatusOK

The interesting thing here is that after some minutes (not sure how many) these do appear in the output even with the code from this PR. I need to re-run elastic-agent from main I have the impression that these appear in the output way faster

[pkoutsovasilis]

I did run again the code of this PR again with both execution modes of the collector (in-process and sub-process) and now all according statuses appear in-time through elastic-agent status --output full so I am gonna choose to unravel this mystery by saying that my previous attempt was on Friday and probably I missed something 🙂

LGTM