Add FleetSim workload mixture evaluation - #2596
Conversation
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Supply Chain Security Report — All Clear
Scanned at |
Signed-off-by: haowu1234 <13258260125@163.com>
805f9ec to
509738a
Compare
github-actions
Bot
assigned
AayushSaini101,
drivebyer,
FAUST-BENCHOU,
ramkrishs,
shraderdm and
WUKUNTAI-0211
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository: 📁
|
| cdf: list[tuple[int, float]] = [] | ||
| last = None | ||
| for idx, total in enumerate(totals, start=1): | ||
| if total == last and idx < n: |
There was a problem hiding this comment.
The duplicate handling records the cumulative probability at the first occurrence rather than the last one. For example, [10, 10, 20] becomes [(10, 1/3), (20, 1)], although the CDF at 10 should be 2/3; [10, 20, 20] also produces duplicate thresholds. Since traces commonly contain repeated token lengths, this can skew the capacity and cost estimates. Could we aggregate equal totals before constructing the empirical CDF and add coverage for both cases?
There was a problem hiding this comment.
Fixed in 5dd0ee6. _empirical_cdf now groups equal token totals before emitting CDF points, so duplicate lengths use the last cumulative probability. Added regression coverage for both [10, 10, 20] and [10, 20, 20]. Local and clean remote make agent-ci-gate CHANGED_FILES="..." both pass.
| weights: dict[str, float], | ||
| cdfs: dict[str, list[tuple[int, float]]], | ||
| description: str, | ||
| ) -> MixtureStressCase: |
There was a problem hiding this comment.
The mixture schema carries model_eligibility and residency, but _case() collapses each workload into a token CDF and the optimizer never evaluates those constraints. This means a "robust" recommendation may combine archetypes that cannot actually run on the same model or placement, even though #2556 requires these constraints to remain hard. Could we either enforce them when building stress cases or avoid returning a feasible recommendation when they cannot be represented?
There was a problem hiding this comment.
Fixed in 5dd0ee6. Mixture stress cases now carry active archetypes plus common model_eligibility / residency; when multiple constrained active archetypes have no common intersection, the case is marked infeasible and robust recommendation is suppressed rather than reporting a feasible CDF-only result. The report/API surface those constraint fields, and tests cover the fail-closed path. Local and clean remote make agent-ci-gate CHANGED_FILES="..." both pass.
Signed-off-by: haowu1234 <13258260125@163.com>
|
This pull request has had no activity from the submitter for 30 days and has been marked as stale. It will be closed in 30 days if no further activity occurs. |
Xunzhuo
added
the
pr/blocked
8 participants
Summary
Closes #2556.
Adds FleetSim workload archetype mixture support for offline capacity experiments:
Production routing/actuation paths are intentionally untouched; this is FleetSim/offline evaluation plumbing.
Validation
git diff --checkpython3 -m pytest tests/test_workload_mixtures.py tests/test_workload_mixture_optimizer.py tests/test_optimizer.py::TestCLISubcommands::test_mixture_optimize tests/test_api.py::TestWorkloadRoutes::test_list_mixture_workloads_returns_three tests/test_api.py::TestJobRoutes::test_submit_mixture_optimize_missing_params_returns_422 tests/test_api.py::TestJobLifecycle::test_mixture_optimize_job_completes_and_has_result -qmake vllm-sr-sim-testmake agent-ci-gate CHANGED_FILES="src/fleet-sim/README.md,src/fleet-sim/data/README.md,src/fleet-sim/data/workload_mixture_burst.json,src/fleet-sim/data/workload_mixture_drift.json,src/fleet-sim/data/workload_mixture_nominal.json,src/fleet-sim/fleet_sim/api/models.py,src/fleet-sim/fleet_sim/api/routes/jobs.py,src/fleet-sim/fleet_sim/api/routes/workloads.py,src/fleet-sim/fleet_sim/api/runner.py,src/fleet-sim/fleet_sim/core/request.py,src/fleet-sim/fleet_sim/data/README.md,src/fleet-sim/fleet_sim/data/workload_mixture_burst.json,src/fleet-sim/fleet_sim/data/workload_mixture_drift.json,src/fleet-sim/fleet_sim/data/workload_mixture_nominal.json,src/fleet-sim/fleet_sim/optimizer/__init__.py,src/fleet-sim/fleet_sim/optimizer/mixture.py,src/fleet-sim/fleet_sim/optimizer/mixture_models.py,src/fleet-sim/fleet_sim/workload/__init__.py,src/fleet-sim/fleet_sim/workload/mixture.py,src/fleet-sim/fleet_sim/workload/mixture_sampling.py,src/fleet-sim/fleet_sim/workload/mixture_validation.py,src/fleet-sim/run_sim.py,src/fleet-sim/tests/test_api.py,src/fleet-sim/tests/test_optimizer.py,src/fleet-sim/tests/test_workload_mixture_optimizer.py,src/fleet-sim/tests/test_workload_mixtures.py"haowu1234:codex/fleetsim-workload-mixturesat509738a74ba074aa0efe6006d60b67892f494007:make agent-ci-gate CHANGED_FILES="..."