Adopt OTel cicd.* semantic conventions incl cicd.* metrics

[cyrille-leclerc]

Add OTel cicd.* semantic conventions including cicd.* metrics.

https://opentelemetry.io/docs/specs/semconv/cicd/cicd-metrics/

TODO

• Boolean toggle to disable generation of older Jenkins metrics that are replaced by the cicd.* metrics.

Testing done

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

---

Lessons learned adopting the OTel CI/CD specs

cicd.pipeline.run.duration

• What are the histogram buckets? We picked 1D, 2D, 4D, 8D, 16D, 32D, 64D, 128D, 256D, 512D, 1024D, 2048D, 4096D, 8192D seconds
• metric attributes
  • cicd.pipeline.name:
    • to protect against metric cardinality explosion, we have allow & deny lists of names reported pipeline duration. Other pipelines are grouped under cicd.pipeline.name=#other#
    • In Jenkins, we have an "id" called "name" and a "display name", we use the "name" for this attribute.
  • cicd.pipeline.run.state: as we record the metric at the end of the execution of the pipeline run, we always have cicd.pipeline.run.duration= finalizing and the real value should be finalized
  • error.type: not yet implemented

cicd.pipeline.run.active

• Metric attributes
  • cicd.pipeline.name:
    • Cardinality protection with allow & deny lists defaulting to cicd.pipeline.name=#other#
    • Is it justified to capture the granularity of this metric per pipeline name?
  • cicd.pipeline.run.state: nothing special to say

cicd.worker.count

• OTel specs say it's an UpDownCounter but implementation in Jenkins is more of a Gauge, a cumulative value captured by a daemon thread.
• Jenkins also has the granularity of tagging build agents by labels (eg macos, gpu, a-special-of a build-agent...) and we also report executor status per label. It's not covered by the OTel CI/CD specs.
• Metric attributes
  • cicd.worker.state
    • The specs give the example of down but the generated java code is offline

cicd.pipeline.run.errors

• Metric attributes
  • cicd.pipeline.name
    • Cardinality protection with allow & deny lists defaulting to cicd.pipeline.name=#other#
    • Is it justified to capture the granularity of this metric per pipeline name?
  • error.type: basic instrumentation: just the Jenkins statuses unstable and failure

cicd.system.errors

• Note implemented yet, we don't differentiate well pipeline errors versus CI/CD system errors.

Other metrics

We need other metrics to report on jenkins health, particularly on build nodes which are holding the build workers

Span attributes

TODO

[kuisathaverat]

About the cardinality protection with allow & deny lists defaulting to cicd.pipeline.name=#other#, Would it be optional? I mean, if I want to grab all my jobs because I do not think the cardinality will be a problem in my case, but I do not want to add my hundred jobs to that allow list, I have an option to choose so that all job names are reported.
Another option that may make more sense is to create groups of jobs, so for example, we group the jobs by team or project or other category I choose.

[cyrille-leclerc]

About the cardinality protection with allow & deny lists defaulting to cicd.pipeline.name=#other#, Would it be optional?

We have implemented with a regular expression. It would be easy to provide a GUI to allow allo pipeline names. Current config to create metrics for all metrics:

otel.instrumentation.jenkins.run.metric.duration.allow_list=.*

[kamphaus]

Anything I can help with?

[cyrille-leclerc]

With pleasure.
From what I remember, what's remaining is:

• Integrate a feature toggle, probably as part of the OTEL_SEMCONV_STABILITY_OPT_IN env var, to enable the old metrics for backward compatibility, the new one to comply with semconv, or both for migration. A challenge is that we our logic should probably be slightly different from existing implementations as we want to be on the old metrics or the mixed mode by default.
• Documentation

I'll be happy to do a knowledge transfer on a video call to help

[ArpanC6]

Hi @cyrille-leclerc, thank you for the detailed notes on adopting
the OTel CI/CD semantic conventions.

I noticed this PR is out-of-date with main and @kamphaus has built
on top of it in PR #1251. A few observations after reviewing both:

1. Cardinality protection — The regex-based allow/deny list
   approach with otel.instrumentation.jenkins.run.metric.duration.allow_list=.*
   as an opt-out is elegant. kuisathaverat's suggestion of grouping
   pipelines by category (team/project) could be a useful enhancement
   on top of this.

2. cicd.worker.count as UpDownCounter vs Gauge — This is an
   interesting tension. Since Jenkins reports executor state via a
   daemon thread polling model rather than event-driven increments/decrements,
   using a Gauge-style ObservableUpDownCounter seems like the right
   pragmatic choice even if the spec says UpDownCounter.

3. cicd.pipeline.run.state — The observation that we always
   record finalizing at metric emission time rather than finalized
   is worth documenting clearly for users.

I would be happy to help with documentation or testing once
the feature toggle implementation is complete.

[kamphaus]

cicd.worker.count as UpDownCounter vs Gauge ... ObservableUpDownCounter seems like the right
pragmatic choice even if the spec says UpDownCounter

The semantic conventions do not make a distinction between UpDownCounter and ObservableUpDownCounter so cicd.worker.count implemented as ObservableUpDownCounter is compliant.