Skip to content

Commit 76f6f4f

Browse files
committed
test: realign the AgentX frontier tests with #736
Pareto frontier from its selected axes without E2E normalized-interactivity gating" — and shipped with "full checks intentionally skipped for this focused change". Master has been red since: 3 unit tests and 3 e2e suites still assert the gating, and the api-doc digest gate was left tripped. This updates the tests to the behaviour that was chosen. No src behaviour changes. Unit: - overview-data: an agentic point dominated on total tokens and slower on E2E is no longer dropped, so the tier-50 read lands on it (8100) instead of coming back null. - useThroughputData: buildGpuGroups keeps an e2e-dominated point, because eligibility is now decided downstream from the selected axes. - api-route-catalog: reviewed overview-data.ts for #736 and updated its digest. Tier reads change value, but the Overview BFF parameters and the OverviewPageData shape are byte-identical — verified by diffing the exported surface against 08e1f7d — so no published contract moved. E2E: - The "Advanced" x-axis popover is gone and all four modes are flat tabs, so the four specs that opened it now click the tab. Agentic also defaults to Interactivity rather than E2E Normalized Interactivity. - The derived-metrics stub moves to beforeEach. Cypress clears intercepts between tests, and these specs relied on the old default fetching during `before` while the stub was alive; React Query then held the result for the spec. Nothing fetches on load now, so a later switch into that mode issued a live request and rendered no chart. - The URL-restored-mode regression test now restores TTFT. It asked for Interactivity, which is the new default, so it could no longer fail. - overlay-optimal-only asserted that overlay points are hidden because trace-less rows cannot join the canonical frontier. They now stand or fall on the selected axes like any other point, and all five fixture configs are non-dominated there — throughput falls monotonically as interactivity rises — so the suite would have asserted five visible either way. It gets a sixth, deliberately dominated overlay config, so Optimal Only has something to drop; making that point non-dominated fails the test. Verified: 3363 unit tests, 598/602 e2e locally. Two local failures are not from this change — zh-pages passes in isolation (page-load timeout under parallel load), and inference-replay fails on a wall-clock animation assertion, imports nothing touched here, and passed CI on #737. 中文:#736 有意移除了端到端归一化交互性的前沿门控,并"刻意跳过完整检查",导致 master 持续为红:3 项单测与 3 个 e2e 套件仍在断言已被移除的门控,API 文档摘要闸门也未更新。 本提交只把测试对齐到既定行为,不改动任何源码行为。单测:概览档位读数不再因总 token 被支配而丢弃该点(改为 8100);buildGpuGroups 保留被 e2e 支配的点;复核 overview-data.ts 后更新摘要(对外参数与响应结构逐字节一致)。E2E:Advanced 弹出菜单已移除,四个模式改为 平铺标签页;派生指标 stub 移至 beforeEach(Cypress 会在测试间清除拦截);URL 恢复模式 的回归测试改用 TTFT(原用 Interactivity 已与新默认值相同而无法失败);overlay-optimal-only 新增一个刻意被支配的配置,否则该断言无论过滤器如何都会通过。
1 parent b367003 commit 76f6f4f

7 files changed

Lines changed: 203 additions & 119 deletions

File tree

packages/app/cypress/e2e/csv-export-overlay.cy.ts

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,10 @@ function readCapturedCsv(): Cypress.Chainable<string> {
3535
describe('Inference CSV export with an unofficial-run overlay', () => {
3636
before(() => {
3737
interceptOverlayRun();
38-
// The agentic default mode is E2E Normalized Interactivity (which suppresses overlays and
39-
// fetches derived metrics) — stub the fetch, then switch to the
40-
// Interactivity mode this suite's overlay assertions rely on.
38+
// Agentic charts default to Interactivity, where the overlay renders. The
39+
// derived-metrics fetch now happens only under E2E Normalized Interactivity,
40+
// so the stub below is a guard against a stray request rather than a
41+
// dependency of this suite.
4142
interceptDerivedAgenticMetrics();
4243
cy.visit(`/inference?unofficialrun=${OVERLAY_RUN_ID}&i_seq=agentic-traces&i_pctl=p90`, {
4344
onBeforeLoad(win) {
@@ -47,12 +48,11 @@ describe('Inference CSV export with an unofficial-run overlay', () => {
4748
},
4849
});
4950
cy.wait('@unofficialRun');
50-
// Interactivity is nested under the Advanced menu on agentic charts.
51-
cy.get('[data-testid="x-axis-mode-advanced"]').click();
52-
cy.get('[data-testid="x-axis-mode-interactivity"]').click();
53-
cy.get('[data-testid="x-axis-mode-advanced"]')
54-
.should('have.attr', 'data-state', 'active')
55-
.and('contain.text', 'Interactivity');
51+
// Every x-axis metric is a top-level tab on agentic charts, and Interactivity
52+
// is the default — clicked anyway so the suite does not depend on that.
53+
cy.get('[data-testid="x-axis-mode-interactivity"]')
54+
.click()
55+
.should('have.attr', 'data-state', 'active');
5656
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should('exist');
5757
});
5858

packages/app/cypress/e2e/overlay-legend-remove.cy.ts

Lines changed: 24 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,10 @@ describe('Official legend X works while an unofficial overlay is loaded', () =>
2222
// Use distinct hardware so the engine-comparison exclusion policy does
2323
// not resolve the official and unofficial rows as one competing family.
2424
interceptOverlayRun({ overlayHardware: 'h100' });
25-
// The agentic default mode is E2E Normalized Interactivity (which suppresses overlays and
26-
// fetches derived metrics) — stub the fetch, then switch to Interactivity
27-
// where the overlay renders.
25+
// Agentic charts default to Interactivity, where the overlay renders. The
26+
// derived-metrics fetch now happens only under E2E Normalized Interactivity,
27+
// so the stub below is a guard against a stray request rather than a
28+
// dependency of this suite.
2829
interceptDerivedAgenticMetrics();
2930
cy.visit(`/inference?unofficialrun=${OVERLAY_RUN_ID}&i_seq=agentic-traces&i_pctl=p90`, {
3031
onBeforeLoad(win) {
@@ -33,16 +34,28 @@ describe('Official legend X works while an unofficial overlay is loaded', () =>
3334
},
3435
});
3536
cy.wait('@unofficialRun');
36-
// Interactivity is nested under the Advanced menu on agentic charts.
37-
cy.get('[data-testid="x-axis-mode-advanced"]').click();
38-
cy.get('[data-testid="x-axis-mode-interactivity"]').click();
37+
// Every x-axis metric is a top-level tab on agentic charts, and Interactivity
38+
// is the default — clicked anyway so the suite does not depend on that.
39+
cy.get('[data-testid="x-axis-mode-interactivity"]')
40+
.click()
41+
.should('have.attr', 'data-state', 'active');
3942
cy.get('[data-testid="chart-figure"]').should('have.length.at.least', 1);
4043
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(
4144
'have.length',
4245
REAL_CONFIGS.length,
4346
);
4447
});
4548

49+
// Cypress clears intercepts between tests, so the derived-metrics stub is
50+
// re-registered per test rather than once in `before`. Until #736 the
51+
// agentic default was E2E Normalized Interactivity, so the fetch happened
52+
// during `before` while the stub was still alive and React Query held the
53+
// result for the rest of the spec. The default no longer fetches, so any
54+
// later switch into that mode issues a fresh request.
55+
beforeEach(() => {
56+
interceptDerivedAgenticMetrics();
57+
});
58+
4659
it('shows official points and an official legend entry initially', () => {
4760
cy.get('[data-testid="inference-chart-display"] svg .dot-group').should(($dots) => {
4861
expect(countVisible($dots), 'visible official points').to.be.greaterThan(0);
@@ -69,10 +82,12 @@ describe('Official legend X works while an unofficial overlay is loaded', () =>
6982
cy.get('[data-testid="inference-chart-display"] svg .dot-group').should(($dots) => {
7083
expect(countVisible($dots), 'visible official points after remove').to.eq(0);
7184
});
72-
// The overlay series is untouched. Optimal Only still hides its trace-less
73-
// points because unofficial rows cannot join the canonical frontier.
85+
// The overlay series is untouched — hiding an official SKU must not disturb
86+
// it. All five overlay points stay visible under Optimal Only: they are
87+
// non-dominated on the interactivity axes, and since #736 lacking persisted
88+
// traces no longer excludes them from the frontier.
7489
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(($pts) => {
75-
expect(countVisible($pts), 'visible overlay X markers').to.eq(0);
90+
expect(countVisible($pts), 'visible overlay X markers').to.eq(REAL_CONFIGS.length);
7691
});
7792
// Inactive row: the hover affordance flips to the "+" restore indicator
7893
// (explicit "clicking the name brings it back"), and the Hide X is gone.
Lines changed: 40 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,31 @@
11
/**
2-
* Unofficial runs do not have persisted request traces, so they cannot join
3-
* the canonical E2E Normalized Interactivity frontier. Optimal Only must hide
4-
* them on every AgentX axis; Show All remains the explicit way to inspect them.
2+
* Optimal Only must filter overlay (unofficial-run) points by the same rule it
3+
* applies to official ones: the Pareto frontier of the *selected* axes.
4+
*
5+
* Until #736 the rule was different — unofficial runs have no persisted request
6+
* traces, so they could never join the canonical E2E Normalized Interactivity
7+
* frontier and Optimal Only hid every one of them. That gating is gone, so an
8+
* overlay point now survives exactly when it is non-dominated on the axes on
9+
* screen, and Show All remains the way to inspect the rest.
510
*/
611
import { interceptDerivedAgenticMetrics, unlockAgenticGate } from '../support/e2e';
712
import {
813
countVisible,
14+
DOMINATED_CONFIG,
915
interceptOverlayRun,
1016
OVERLAY_RUN_ID,
1117
REAL_CONFIGS,
1218
} from '../support/overlay-fixtures';
1319

14-
describe('Overlay points follow canonical Optimal Only policy (agentic interactivity)', () => {
20+
// The five real configs are all non-dominated on the interactivity axes, so a
21+
// sixth, deliberately dominated point is what gives Optimal Only something to
22+
// remove. Without it this suite would assert the same count either way and pass
23+
// no matter what the filter did.
24+
const OVERLAY_CONFIGS = [...REAL_CONFIGS, DOMINATED_CONFIG];
25+
26+
describe('Overlay points follow Optimal Only on the selected axes', () => {
1527
before(() => {
16-
interceptOverlayRun();
17-
// The agentic default mode is E2E Normalized Interactivity (which suppresses overlays and
18-
// fetches derived metrics) — stub the fetch, then switch to the
19-
// Interactivity mode this suite is about.
28+
interceptOverlayRun({ overlayConfigs: OVERLAY_CONFIGS });
2029
interceptDerivedAgenticMetrics();
2130
cy.visit(`/inference?unofficialrun=${OVERLAY_RUN_ID}&i_seq=agentic-traces&i_pctl=p90`, {
2231
onBeforeLoad(win) {
@@ -25,46 +34,53 @@ describe('Overlay points follow canonical Optimal Only policy (agentic interacti
2534
},
2635
});
2736
cy.wait('@unofficialRun');
28-
// Interactivity is nested under the Advanced menu on agentic charts.
29-
cy.get('[data-testid="x-axis-mode-advanced"]').click();
30-
cy.get('[data-testid="x-axis-mode-interactivity"]').click();
37+
// Every x-axis metric is a top-level tab on agentic charts, and Interactivity
38+
// is the default — clicked anyway so the suite does not depend on that.
39+
cy.get('[data-testid="x-axis-mode-interactivity"]')
40+
.click()
41+
.should('have.attr', 'data-state', 'active');
3142
cy.get('[data-testid="chart-figure"]').should('have.length.at.least', 1);
32-
cy.get('[data-testid="x-axis-mode-advanced"]')
33-
.should('have.attr', 'data-state', 'active')
34-
.and('contain.text', 'Interactivity');
43+
// All six are rendered; visibility is what Optimal Only changes.
3544
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(
3645
'have.length',
37-
REAL_CONFIGS.length,
46+
OVERLAY_CONFIGS.length,
3847
);
3948
});
4049

41-
it('hides trace-less overlay points in the default Optimal Only view', () => {
50+
// Cypress clears intercepts between tests, so the derived-metrics stub is
51+
// re-registered per test rather than once in `before`. Until #736 the agentic
52+
// default was E2E Normalized Interactivity, so the fetch happened during
53+
// `before` while the stub was still alive and React Query held the result for
54+
// the rest of the spec. The default no longer fetches.
55+
beforeEach(() => {
56+
interceptDerivedAgenticMetrics();
57+
});
58+
59+
it('drops only the dominated overlay point in the default Optimal Only view', () => {
4260
cy.get('#scatter-hide-non-optimal').should('have.attr', 'data-state', 'checked');
43-
// The deterministic derived-metric stub puts all five official rows on the
44-
// canonical frontier.
4561
cy.get('[data-testid="inference-chart-display"] svg .dot-group').should(($dots) => {
4662
expect(countVisible($dots), 'visible official points').to.eq(REAL_CONFIGS.length);
4763
});
48-
// Overlay rows have no persisted trace ids and therefore no canonical
49-
// frontier membership.
64+
// Five of six survive: every real config is non-dominated on these axes, and
65+
// the overlay's trace-less rows are no longer excluded for lacking traces.
5066
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(($pts) => {
51-
expect(countVisible($pts), 'visible overlay X markers').to.eq(0);
67+
expect(countVisible($pts), 'visible overlay X markers').to.eq(REAL_CONFIGS.length);
5268
});
5369
});
5470

5571
it('shows all overlay points when Optimal Only is turned off', () => {
5672
cy.get('#scatter-hide-non-optimal').click();
5773
cy.get('#scatter-hide-non-optimal').should('have.attr', 'data-state', 'unchecked');
5874
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(($pts) => {
59-
expect(countVisible($pts), 'visible overlay X markers').to.eq(REAL_CONFIGS.length);
75+
expect(countVisible($pts), 'visible overlay X markers').to.eq(OVERLAY_CONFIGS.length);
6076
});
6177
});
6278

63-
it('re-hides trace-less overlay points when Optimal Only is re-enabled', () => {
79+
it('re-drops the dominated overlay point when Optimal Only is re-enabled', () => {
6480
cy.get('#scatter-hide-non-optimal').click();
6581
cy.get('#scatter-hide-non-optimal').should('have.attr', 'data-state', 'checked');
6682
cy.get('[data-testid="inference-chart-display"] svg .unofficial-overlay-pt').should(($pts) => {
67-
expect(countVisible($pts), 'visible overlay X markers').to.eq(0);
83+
expect(countVisible($pts), 'visible overlay X markers').to.eq(REAL_CONFIGS.length);
6884
});
6985
});
7086
});

0 commit comments

Comments
 (0)