fix(sa-bench): fix rollup key mapping for p99 latencies, token counts, and ISL/OSL - #241
fix(sa-bench): fix rollup key mapping for p99 latencies, token counts, and ISL/OSL#241arpera wants to merge 5 commits into
Conversation
…, and ISL/OSL Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
|
|
||
| valid_low_concurrency = { | ||
| "model_id": "GLM-5-FP8", | ||
| "random_input_len": 8192, |
There was a problem hiding this comment.
hold up. they've removed isl and osl from the spec ? this a load bearing metric for fixed isl osl workloads. the conforms with agentic workloads where total tokens matter more. let me confirm from the latest ci run from inferencex.
There was a problem hiding this comment.
yeah this as been moved from random_input_len -> isl
https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28916955742/artifacts/8164384297
There was a problem hiding this comment.
Hi Kaunil, probably I misunderstand something but I did this PR only targeting sa-bench implementation from srt-slurm (not InferenceX's sa-bench). I suppose InferenceX has it's own implementation of sa-bench and this implementation differs from srt-slurm's one (particularly InferenceX's one has isl/osl field in results json file, but srt-slurm's one does not have them). If I understand right then srt-slurm's implementation of sa-bench does not have isl filed in json file. So, my point is that in this PR I am attempt to fix srt-slurm's sa-bench results parsing not InferenceX's sa-bench. Could you tell me please what I should do then?
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
2 participants
Rollup key mapping is wrong for
sa-bench. Some of the fields were affected by this (for example, ISL/OSL) resulting in beingNonein the final json statistics file.Testing Done
sa-benchafter applying this patch showed noNonefields in json statistics file.make check-- no issues found