fix(agentic): give KV cache series a real per-engine identity

docs/data-pipeline.md

@@ -89,6 +89,18 @@ AIPerf defines the `server_metrics_export.json` envelope, but labels such as wor
 
 Adapters are selected from the benchmark's canonical framework, and per-worker series are only emitted for disaggregated configs with a recognized adapter. Unknown orchestrators and non-disaggregated configs retain their aggregate-only series; roles are never guessed from ports or metric names. The frontend only consumes the canonical source identity and never interprets orchestrator-native labels.
 
+### Logical Engines vs Raw Series
+
+A raw series in the blob is one `(scrape endpoint × phase block × label set)` tuple, which is **not** the same as one engine. The KV-cache chart needs one entry per _logical engine_ — one KV pool — so `compute-chart-series.ts` groups series by their Prometheus label set (`seriesIdentityKey`) rather than emitting one entry per raw series. Three kinds of duplication collapse there:
+
+- **Phase blocks.** The warmup and profiling blocks each carry their own series for the same engine, so keying by scrape instant unions them into one continuous line. The two blocks' first/last bounds can look overlapping — the profiling series often emits a single boundary sample and then gaps until warmup ends — but they never share a scrape instant, so the union neither drops nor double-counts a sample.
+- **Mirrored API-server frontends.** vLLM run with several API servers exposes the _same_ engine set on every `/metrics` endpoint, a few hundred ms apart. Identity ignores `endpoint_url` (Prometheus treats the label set as the series), and the endpoint covering the most wall-clock wins — merging the mirrors instead would interleave near-duplicate samples and halve the effective span of the frontend's fixed-width rolling average. Endpoints are only fused when their values actually agree; same-label endpoints whose readings diverge are independent replicas behind a router, and are kept as separate engines so none is silently discarded.
+- **Intra-engine shard ranks.** `tp_rank` / `pp_rank` / `ep_rank` / `moe_ep_rank` shard one pool and all report the same utilization, so they are excluded from the identity and averaged. `engine` / `engine_idx` / `dp_rank` are _not_ excluded — those do name distinct pools.
+
+The cluster average is then a mean across those logical engines on the union of their scrape instants, with each engine holding its last sample until its next one and contributing only inside its own observed window (and only while that sample is fresher than 5× the engine's own median scrape gap, so a reporting hole drops the engine out of the mean rather than pinning it to a stale reading). Grouping on an exact `start_ns` instead would average whichever engines happened to share that nanosecond — on a disaggregated run that alternates between "prefill only" and "decode only" and reads as a full-scale sawtooth.
+
+Engines are ordered by role, then numeric rank, then worker — never by the composed display string, which would sort `"decode 10"` before `"decode 2"` and scramble DP ranks. The role is only shown when engines actually differ in role, so an aggregated deployment reads `DP 0…DP 3` rather than `decode 0…decode 3`.
+
 ### Agentic Dataset Provenance
 
 AIPerf exports public-dataset provenance in `metadata.dataset`, including the Hugging Face dataset ID. InferenceX preserves that object as `dataset` on each agentic aggregate benchmark row. During benchmark ingest, `ingest-ci-run.ts` derives the dashboard slug from `hf_dataset_name` (for example, `semianalysisai/cc-traces-weka-062126` becomes `cc-traces-weka-062126`) and upserts `run_datasets` for the workflow run.

packages/app/cypress/e2e/agentic-point-time-series.cy.ts

@@ -353,3 +353,63 @@ describe('Agentic point orchestrator metric sources', () => {
     cy.get('[data-testid="throughput-series-decode"]').should('have.attr', 'aria-pressed', 'false');
   });
 });
+
+const engineSeries = (engineLabel: string, value: number) => ({
+  engineLabel,
+  points: [
+    { t: 0, value },
+    { t: 1, value: value + 0.05 },
+  ],
+});
+
+describe('Agentic point per-engine KV overlay', () => {
+  beforeEach(() => {
+    cy.intercept('GET', '/api/v1/trace-histograms*', { body: {} });
+    cy.intercept('GET', '/api/v1/benchmark-siblings*', { statusCode: 404 });
+    cy.intercept('GET', '/api/v1/request-timeline*', { statusCode: 404 });
+    cy.intercept('GET', '/api/v1/trace-server-metrics*', {
+      body: {
+        meta: pointMeta,
+        startNs: 0,
+        endNs: 2_000_000_000,
+        durationS: 2,
+        timeslicesCount: 2,
+        kvCacheUsage: [
+          { t: 0, value: 0.3 },
+          { t: 1, value: 0.35 },
+        ],
+        prefixCacheHitRate: [],
+        queueDepth: [],
+        promptTokensBySource: {},
+        prefillTps: [],
+        decodeTps: [],
+        prefixCacheHitsTps: [],
+        hostKvCacheUsage: [],
+        // Bare DP ranks plus a role-qualified engine, as the ETL emits for a
+        // disaggregated run where the decode worker reports no rank label.
+        kvCacheUsageByEngine: [
+          engineSeries('0', 0.2),
+          engineSeries('1', 0.4),
+          engineSeries('decode', 0.6),
+        ],
+        metricSources: [],
+      },
+    });
+    cy.visit('/inference/agentic/206885', { onBeforeLoad: unlockAgenticGate });
+  });
+
+  it('draws one legend entry per engine and leaves named engines unprefixed', () => {
+    cy.contains('svg', 'KV cache (%)')
+      .first()
+      .within(() => {
+        cy.contains('text', 'DP 0').should('be.visible');
+        cy.contains('text', 'DP 1').should('be.visible');
+        // Already self-describing — must NOT come out as "DP decode".
+        cy.contains('text', 'decode').should('be.visible');
+        cy.contains('text', 'DP decode').should('not.exist');
+        cy.contains('text', 'Avg').should('be.visible');
+        // One line per engine plus the average, and no duplicate chips.
+        cy.get('path[fill="none"]').should('have.length', 4);
+      });
+  });
+});

packages/app/src/app/api/v1/trace-server-metrics/route.ts

@@ -12,6 +12,15 @@ import { idQueryRoute } from '../id-routes';
 
 export const dynamic = 'force-dynamic';
 
+// The slow path re-parses `server_metrics_json_gz`, which reaches 448 MB
+// compressed (several GB of JSON) on the largest agentic points and takes
+// minutes to stream. That only happens for rows whose stored `chart_series`
+// predates the current version — i.e. after a version bump, until
+// `db:backfill-chart-series` drains them — but the platform default would
+// cut those requests off mid-parse and the blob cache only populates on
+// success, so every visitor would re-pay. Ask for the maximum instead.
+export const maxDuration = 300;
+
 // Key derived from TRACE_SERVER_METRICS_VERSION (governs chart_series plus
 // the separately queried point-metadata payload).
 // The blob cache is write-once with no post-backfill purge, so the

packages/app/src/components/inference/agentic-point/server-metric-cards.tsx

@@ -62,6 +62,15 @@ const DP_RANK_PALETTE = [
   '#eab308',
 ];
 
+/**
+ * Bare DP ranks read as "DP 3"; disaggregated runs get role- or worker-
+ * qualified labels from the ETL ("decode", "prefill 0", "0 (a01a)") that
+ * already name themselves, so those pass through untouched.
+ */
+function engineSeriesName(engineLabel: string): string {
+  return /^\d+$/u.test(engineLabel) ? `DP ${engineLabel}` : engineLabel;
+}
+
 export function KvCacheUtilizationCard({ sliced }: { sliced: SlicedServerSeries }) {
   return (
     <ExpandableChart
@@ -77,21 +86,29 @@ export function KvCacheUtilizationCard({ sliced }: { sliced: SlicedServerSeries
         // than one, draw one line per rank in distinct colors so
         // load skew is visible at a glance; cluster-average sits on
         // top in white so it stands out.
-        const perEngine = serverSeries.kvCacheUsageByEngine ?? [];
-        const hasPerEngine = perEngine.length > 1;
+        const allEngines = serverSeries.kvCacheUsageByEngine ?? [];
+        // Decide off the point's own engine count, not the phase-sliced one:
+        // this also drives the average line's name, color and stroke, so
+        // keying it to the slice would make the chart change identity when
+        // you switch between the Warmup and Profiling tabs.
+        const hasPerEngine = allEngines.length > 1;
+        // Colors come from the unsliced position so a rank keeps its color
+        // across phases; engines with no points in this phase are dropped
+        // afterwards so they don't take a legend slot with no line.
+        const perEngine = allEngines
+          .map((e, i) => ({
+            name: engineSeriesName(e.engineLabel),
+            data: rollingAverage(e.points, 50),
+            color: DP_RANK_PALETTE[i % DP_RANK_PALETTE.length]!,
+            // Thin + translucent so the Avg line on top reads as
+            // the headline number, not just one more series.
+            strokeWidth: 1,
+            strokeOpacity: 0.5,
+          }))
+          .filter((s) => s.data.length > 0);
         // Render order matters: per-engine first → average drawn on top.
         const series = [
-          ...(hasPerEngine
-            ? perEngine.map((e, i) => ({
-                name: `DP ${e.engineLabel}`,
-                data: rollingAverage(e.points, 50),
-                color: DP_RANK_PALETTE[i % DP_RANK_PALETTE.length]!,
-                // Thin + translucent so the Avg line on top reads as
-                // the headline number, not just one more series.
-                strokeWidth: 1,
-                strokeOpacity: 0.5,
-              }))
-            : []),
+          ...(hasPerEngine ? perEngine : []),
           {
             name: hasHost
               ? 'Chip HBM (avg n=50)'