[Feature Request] Support Prometheus exemplars on Core metrics

[TheCodeWrangler]

Status: proposal / request for comment

Motivation

Core's Prometheus metrics use a deliberately low-cardinality label set: namespace, task_queue, workflow_type, activity_type, poller_type. Per-execution identifiers — run_id, workflow_id, activity_id — are absent, correctly, since as labels they would multiply series count by every execution the worker handles.

The cost is that latency histograms are unactionable at the tail. When temporal_workflow_task_execution_latency p99 jumps, the metrics establish that something was slow and offer no path to what. Exemplars are the OpenMetrics feature for exactly this: a bounded number of high-cardinality sample pointers attached to buckets, stored outside the series and ignored by aggregation.

For Core this is unusually cheap, because the identifiers are already in scope at the record sites. In crates/sdk-core/src/worker/workflow/managed_run.rs, the call recording workflow task latency sits two lines from an existing use of self.wfm.machines.run_id. No propagation, no changes to the worker/client boundary.

Proposal

1. An exemplars option on PrometheusExporterOptions, default off. When off, exposition output is byte-identical to today.
2. Core attaches run_id where it already has it — starting with the workflow task and end-to-end latency histograms.
3. Lang SDKs can supply exemplar labels for custom metrics, via an additive trait method with a defaulted body, so existing CoreMeter implementations compile unchanged and ignore exemplars.
4. OpenMetrics exposition, content-negotiated. Exemplars have no representation in text format 0.0.4, so emit OpenMetrics on Accept: application/openmetrics-text when the option is on; every other request keeps the current TextEncoder path.

Non-goals: trace_id exemplars — Core has no distributed tracing, tracing-opentelemetry is not a dependency and SpanContext/trace_id appear nowhere in crates/common/src/telemetry/, so SDKs would supply these via item 3; gauges, which OpenMetrics disallows; and the OTel and buffered meters, which ignore exemplars via the defaulted method.

Implementation notes

The prometheus crate's histogram type has no exemplar slot, and we depend on it with default-features = false, so there is no Exemplar type in scope either. Cheapest path is a side table on the custom Registry in prometheus_meter.rs, keyed by family name, label set and bucket index, plus a small OpenMetrics encoder. Switching to prometheus-client would provide both natively but means rewriting ~900 lines, including the custom registry that permits identical names with differing label sets.

Exemplar label validation (OpenMetrics: name charset, combined length ≤128 UTF-8 characters) belongs in Core so there is one implementation for all SDKs. Invalid exemplars get dropped with a warning and the observation still recorded — instrumentation must not break the caller.

What I would like decided

1. Which internal metrics get automatic exemplars, and carrying which identifier?
2. Should automatic run_id exemplars be implied by exemplars = true, or be a separate opt-in? Some operators will want the mechanism for their own custom metrics only.
3. OpenMetrics text, or enable the prometheus crate's protobuf feature instead? Protobuf avoids hand-writing an encoder but is not widely enabled on scrapers.

Two smaller notes: exemplar storage is still feature-gated on the Prometheus server, so operators must opt in there as well; and OpenMetrics requires counters to end in _total, which conflicts with counters_total_suffix defaulting to false — simplest is to force the suffix on OpenMetrics responses only.

[Sushisource]

@TheCodeWrangler What's the actual problem you're trying to solve here? This LLM generated summary of how we could add exemplars is vaguely sensible (though maybe the most important part, the threshold for choosing an automatic exemplar, is left out).

If you can describe (in your own words) how this ties to a specific problem or use case you have, that would help. Otherwise I'll close this. Thanks.