Skip to content

test: realign the AgentX frontier tests with #736 / 将 AgentX 前沿相关测试对齐 #736 - #742

Merged
Oseltamivir merged 1 commit into
masterfrom
fix/agentx-frontier-test-drift
Aug 15, 2026
Merged

test: realign the AgentX frontier tests with #736 / 将 AgentX 前沿相关测试对齐 #736#742
Oseltamivir merged 1 commit into
masterfrom
fix/agentx-frontier-test-drift

Conversation

@Oseltamivir

@Oseltamivir Oseltamivir commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Rebased onto b367003 (post-#743). #743's own body cites this PR as "the exact test-only realignment" for the drift it hit.

Summary

master went red at #736. That PR removed E2E normalized-interactivity frontier gating on purpose — its own summary says "compute each Pareto frontier from its selected axes without E2E normalized-interactivity gating" — and merged with "full checks intentionally skipped for this focused change". The tests asserting the old gating were never updated.

Bisected: 08e1f7d green, 7fded35 (#736) red. This PR changes no src behaviour — it moves the tests to the behaviour that was chosen. Two later PRs (#737, #743) hit the same failures and merged past them.

Unit

Test Now asserts
overview-data A point dominated on total tokens and slower on E2E is no longer dropped, so the tier-50 read lands on it (8100) instead of null
useThroughputData buildGpuGroups keeps an e2e-dominated point; eligibility is decided downstream from the selected axes

The overview-data.ts API-doc digest was part of this PR until the rebase. #743 changed that file again and re-reviewed the entry itself, so api-route-catalog.ts is no longer touched here.

E2E

  • The "Advanced" popover is gone and all four x-axis modes are flat tabs, so the four specs that opened it click the tab instead. Agentic also defaults to Interactivity now, not E2E Normalized Interactivity.
  • The derived-metrics stub moves to beforeEach. Cypress clears intercepts between tests; these specs relied on the old default fetching during before while the stub was alive, after which React Query held the result for the whole spec. Nothing fetches on load anymore, so a later switch into that mode issued a live request and rendered no chart at all.
  • The URL-restored-mode regression test now restores TTFT. It restored Interactivity — the new default — so it could no longer detect the clobber it exists to catch.
  • overlay-optimal-only needed a new fixture point. It asserted overlay points are hidden because trace-less rows cannot join the canonical frontier. They now stand or fall on the selected axes like any other point, and all five fixture configs are non-dominated there — throughput falls monotonically as interactivity rises — so the suite would have asserted five visible whatever the filter did. It gets a sixth, deliberately dominated overlay config (DOMINATED_CONFIG), giving Optimal Only something to drop.

interceptOverlayRun's new overlayConfigs option defaults to REAL_CONFIGS, so the other four consumers are untouched.

Verification (re-run on b367003)

  • Unit: 3372 passed, 173 files.
  • E2E: 600/602 locally; the four repaired specs 30/30 at retries=0, re-run after rebuilding against feat(data): preserve additive append-only recipe variants #743 — which touched overlay and chart consumers, so this was re-verified rather than assumed.
  • Mutation-checked the one new assertion: making DOMINATED_CONFIG non-dominated fails drops only the dominated overlay point… (1 visible vs 5 expected).
  • lint / fmt / typecheck clean.
  • The pre-rebase revision was green in CI on all 8 browser shards.

The two local e2e failures are not from this change: inference-replay asserts a scrubber advances after a wall-clock cy.wait(800) — it imports nothing touched here and passed CI. (zh-pages failed once under parallel load and passes on re-run.)

Note for @cquil11

#736 left restrictAgenticPointsToE2eFrontier in the tree with zero callers, and its own tests still pass, so it now reads as live policy. Separately, MODELS.md in the InferenceX repo still documents the North-star Pareto policy as binding ("a point cannot appear on any AgentX Pareto frontier unless it is both a North Star winner and non-dominated on the selected chart"), which the app no longer enforces. Neither is touched here — deleting dead code and reconciling that doc are your calls, not a test fix's.

中文说明

已 rebase 到 b367003#743 之后)。#743 的说明中亦将本 PR 称为"针对该漂移的纯测试对齐修复"。

master#736 起转红。该 PR 有意移除了端到端归一化交互性的前沿门控(其摘要即为"各 Pareto 前沿仅依据所选坐标轴计算,不再受端到端归一化交互性门控约束"),且合并时"刻意跳过完整检查",但断言旧门控的测试未随之更新。已二分定位:08e1f7d 通过,7fded35#736)失败。本 PR 不改动任何 src 行为,只是把测试对齐到既定行为;此后 #737#743 均遇到同一组失败并带红合并。

单测:概览档位读数不再因总 token 被支配且 E2E 较慢而丢弃该点,tier-50 读数落在 8100 而非 nullbuildGpuGroups 保留被 e2e 支配的点。overview-data.ts 的 API 文档摘要原本也在本 PR 范围内,但 #743 已再次修改该文件并重新复核了该条目,故 api-route-catalog.ts 在此不再改动

E2E:Advanced 弹出菜单已移除,四个模式改为平铺标签页,四个原先打开该菜单的套件改为直接点击标签页;派生指标 stub 移至 beforeEach(Cypress 会在测试间清除拦截,这些套件此前依赖旧默认值在 before 期间完成抓取并由 React Query 缓存,现在加载时不再抓取,后续切换会发出真实请求并导致图表无法渲染);URL 恢复模式的回归测试改用 TTFT;overlay-optimal-only 新增一个刻意被支配的配置 DOMINATED_CONFIG,否则五个真实配置在交互性坐标轴上均非被支配,该断言无论过滤器如何都会通过。interceptOverlayRun 新增的 overlayConfigs 默认为 REAL_CONFIGS,其余四处调用不受影响。

验证(已在 b367003 上重跑):单测 3372 项全部通过;本地 e2e 600/602,四个修复套件在 retries=0 下 30/30——因 #743 改动了叠加层与图表消费方,此处为重新构建后实测而非沿用旧结论;新断言已通过变异测试验证;lint/fmt/typecheck 均通过;rebase 前的版本在 CI 全部 8 个浏览器分片上均为绿色。两项本地失败与本 PR 无关:inference-replay 依赖真实时钟的动画断言且未引用本 PR 改动的任何内容,在 CI 中通过(zh-pages 仅在并行负载下偶发超时,重跑通过)。

另请 @cquil11 注意:#736 之后 restrictAgenticPointsToE2eFrontier 仍留在代码中但已无任何调用方,其自身测试仍通过,容易被误读为仍在生效的策略;此外 InferenceX 仓库的 MODELS.md 仍将 North-star Pareto 政策记为强制约束,而应用已不再执行。这两点本 PR 均未触碰——删除死代码与同步该文档应由你决定。


Note

Low Risk
Only test and Cypress fixture changes; production code is untouched.

Overview
Test-only alignment after #736 removed E2E normalized-interactivity frontier gating and reshaped the agentic chart UI. No src behavior changes in this PR.

Unit tests now expect Pareto and tier logic on the selected axes only: buildGpuGroups keeps e2e-dominated agentic points, and overview AgentX tier reads can land on points that are dominated on total tokens / slower on E2E (8100 instead of null).

Cypress drops the Advanced popover flow—four x-axis modes are flat tabs; agentic defaults to Interactivity (not E2E Normalized Interactivity). Derived-metrics intercepts move to beforeEach because the default no longer fetches on load and Cypress clears intercepts between tests. Overlay specs assert Optimal Only hides only dominated overlay points (not all trace-less overlays); DOMINATED_CONFIG and overlayConfigs on interceptOverlayRun make that test meaningful. URL-restore regression uses TTFT so it can still detect clobbering against the new default.

Reviewed by Cursor Bugbot for commit 6094873. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
inferencemax-app Ready Ready Preview Aug 15, 2026 6:03am

Request Review

#736 removed E2E normalized-interactivity gating on purpose — "compute each
Pareto frontier from its selected axes without E2E normalized-interactivity
gating" — and shipped with "full checks intentionally skipped for this focused
change". Master went red: unit tests and e2e suites still asserted the gating.
This updates the tests to the behaviour that was chosen. No src behaviour
changes.

Unit:

- overview-data: an agentic point dominated on total tokens and slower on E2E is
  no longer dropped, so the tier-50 read lands on it (8100) instead of coming
  back null.
- useThroughputData: buildGpuGroups keeps an e2e-dominated point, because
  eligibility is now decided downstream from the selected axes.

The api-route-catalog digest for overview-data.ts was part of this fix until the
rebase onto #743, which changed that file again and re-reviewed the entry. It is
no longer touched here.

E2E:

- The "Advanced" x-axis popover is gone and all four modes are flat tabs, so the
  four specs that opened it now click the tab. Agentic also defaults to
  Interactivity rather than E2E Normalized Interactivity.
- The derived-metrics stub moves to beforeEach. Cypress clears intercepts
  between tests, and these specs relied on the old default fetching during
  `before` while the stub was alive; React Query then held the result for the
  spec. Nothing fetches on load now, so a later switch into that mode issued a
  live request and rendered no chart.
- The URL-restored-mode regression test now restores TTFT. It asked for
  Interactivity, which is the new default, so it could no longer fail.
- overlay-optimal-only asserted that overlay points are hidden because
  trace-less rows cannot join the canonical frontier. They now stand or fall on
  the selected axes like any other point, and all five fixture configs are
  non-dominated there — throughput falls monotonically as interactivity rises —
  so the suite would have asserted five visible either way. It gets a sixth,
  deliberately dominated overlay config, so Optimal Only has something to drop;
  making that point non-dominated fails the test.

Verified on b367003: 3372 unit tests, 600/602 e2e locally, and the four repaired
specs 30/30 at retries=0. The two local e2e failures are inference-replay, which
asserts a scrubber advances after a wall-clock cy.wait(800); it imports nothing
touched here and passes in CI.

中文:#736 有意移除了端到端归一化交互性的前沿门控,并"刻意跳过完整检查",导致 master
转红:单测与 e2e 套件仍在断言已被移除的门控。本提交只把测试对齐到既定行为,不改动任何
源码行为。单测:概览档位读数不再因总 token 被支配而丢弃该点(改为 8100);
buildGpuGroups 保留被 e2e 支配的点。overview-data.ts 的 API 文档摘要原本也在本次修复
范围内,但在 rebase 到 #743 后,该 PR 已再次修改此文件并重新复核了该条目,故此处不再
改动。E2E:Advanced 弹出菜单已移除,四个模式改为平铺标签页;派生指标 stub 移至
beforeEach(Cypress 会在测试间清除拦截);URL 恢复模式的回归测试改用 TTFT(原用
Interactivity 已与新默认值相同而无法失败);overlay-optimal-only 新增一个刻意被支配的
配置,否则该断言无论过滤器如何都会通过。已在 b367003 上验证:单测 3372 项通过,本地
e2e 600/602,四个修复套件在 retries=0 下 30/30。
@Oseltamivir
Oseltamivir force-pushed the fix/agentx-frontier-test-drift branch from 76f6f4f to 6094873 Compare August 15, 2026 06:01
@Oseltamivir
Oseltamivir merged commit 0786d2b into master Aug 15, 2026
24 checks passed
@Oseltamivir
Oseltamivir deleted the fix/agentx-frontier-test-drift branch August 15, 2026 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant