Initial version of the OpenTelemetry System content pack

[AlexanderWert]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

Closes #14304

[kpollich]

Looks like the build is failing because there's no codeowner specified in .github/CODEOWNERS for this content pack.

[kpollich]

I went through the OTel onboarding screen here:

I ran this on a default Ubuntu VM through Multipass.

I checked out the PR branch here using gh pr checkout and built the integration, e.g.

$ cd packages/system_otel
$ elastic-package build

Then, I generated an API key, and uploaded this content pack directly to a serverless observability project via cURL e.g.

# Find the built zip in your integration repo's `build` directory
$ cd build/packages

$ curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' -H 'Authorization: ApiKey <redacted>' --data-binary @system_otel-1.0.0.zip https://otel-content-pack-test-ff3198.kb.us-east-1.aws.elastic.cloud/api/fleet/epm/packages

I waited a while and went through the provided dashboards. There are some expected errors in the cloud visualizations, but some other ones related to data views that I'm not sure on:

Screen.Recording.2025-07-02.at.9.33.16.AM.mov

[kpollich]

Looks like the recent commit fixed my dataview issue in serverless

In testing against my local dev cluster, though, my OTel data seems to be in the wrong index:

Here's my otel.yml on my multipass VM after running the onboarding script for host logs/metrics. Note I did have to update the IP for my elasticsearch exporter since my VM can't resolve localhost:9200 for the ES snapshot that runs on my local machine.

receivers:
  # Receiver for platform specific log files
  filelog/platformlogs:
    include: [ /var/log/*.log ]
    retry_on_failure:
      enabled: true
    start_at: end
    storage: file_storage
#   start_at: beginning

  # Receiver for CPU, Disk, Memory, and Filesystem metrics
  hostmetrics/system:
    collection_interval: 30s
    scrapers:
      disk:
      filesystem:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      # process scraper is disabled for now: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/39423
      #process:
      #  mute_process_exe_error: true
      #  mute_process_io_error: true
      #  mute_process_user_error: true
      #  metrics:
      #    process.threads:
      #      enabled: true
      #    process.open_file_descriptors:
      #      enabled: true
      #    process.memory.utilization:
      #      enabled: true
      #    process.disk.operations:
      #      enabled: true
      network:
      processes:
      load:

extensions:
  file_storage:
    directory: /home/ubuntu/elastic-distro-9.0.3-linux-arm64/data/otelcol

processors:
  elasticinframetrics:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: false
        host.arch:
          enabled: true
        host.ip:
          enabled: true
        host.mac:
          enabled: true
        host.cpu.vendor.id:
          enabled: true
        host.cpu.family:
          enabled: true
        host.cpu.model.id:
          enabled: true
        host.cpu.model.name:
          enabled: true
        host.cpu.stepping:
          enabled: true
        host.cpu.cache.l2.size:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true
  attributes/dataset:
    actions:
      - key: event.dataset
        from_attribute: data_stream.dataset
        action: upsert
  resource/process:
    attributes:
      - key: process.executable.name
        action: delete
      - key: process.executable.path
        action: delete

exporters:
  # Exporter to send logs and metrics to Elasticsearch
  elasticsearch/ecs:
    endpoints: ["http://192.168.4.117:9200"]
    api_key: RjVHWHk1Y0JVSkNybnZtNE9MNmU6LTNLVEpRVmFPZ0tfelhjamdxakZvdw==
    mapping:
      mode: ecs
    logs_dynamic_index:
      enabled: true
    metrics_dynamic_index:
      enabled: true
    traces_dynamic_index:
      enabled: true
  elasticsearch/otel:
    endpoints: [ "http://192.168.4.117:9200" ]
    api_key: RjVHWHk1Y0JVSkNybnZtNE9MNmU6LTNLVEpRVmFPZ0tfelhjamdxakZvdw==
    mapping:
      mode: otel
    logs_dynamic_index:
      enabled: true
    metrics_dynamic_index:
      enabled: true
    traces_dynamic_index:
      enabled: true

service:
  extensions: [file_storage]
  pipelines:
    metrics/hostmetrics:
      receivers: [hostmetrics/system]
      processors: [elasticinframetrics, resourcedetection, attributes/dataset, resource/process]
      exporters: [elasticsearch/ecs]
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]

[AlexanderWert]

n testing against my local dev cluster, though, my OTel data seems to be in the wrong index:
...
Here's my otel.yml on my multipass VM after running the onboarding script for host logs/metrics. Note I did have to update the IP for my elasticsearch exporter since my VM can't resolve localhost:9200 for the ES snapshot that runs on my local machine.

@kpollich This otel.yml still uses the elasticinframetrics processor and outputs the metrics in ECS format, that's why. You'd need to change the pipelines:

• remove elasticinframetrics processor from the pipeline
• change exporter from elasticsearch/ecs to elasticsearch/otel

[rogercoll]

LGTM, just a small comment regarding the cpu utilization line tag

[kpollich]

Thanks @AlexanderWert - I made those changes and everything looks good to me in my local cluster now.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
99.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 6211ef3

History

• 💚 Build #28084 succeeded 0a8ba92
• 💚 Build #28057 succeeded d9e8d4b
• 💚 Build #28032 succeeded ab52668
• 💚 Build #28025 succeeded 39103cf
• 💚 Build #28019 succeeded 8bfd772
• 💚 Build #27991 succeeded 36932f8

[elastic-vault-github-plugin-prod]

Package system_otel - 0.1.0 containing this change is available at https://epr.elastic.co/package/system_otel/0.1.0/