feat(bench): Add opt-out toggle for community benchmark upload#1052
Open
jabagawee wants to merge 1 commit into
Open
feat(bench): Add opt-out toggle for community benchmark upload#1052jabagawee wants to merge 1 commit into
jabagawee wants to merge 1 commit into
Conversation
Throughput benchmarks upload results to omlx.ai unconditionally at the end of every run, including a stable per-device hash and the model name (which can be a private/local model id). There is no UI toggle, env var, or request flag to disable it. This commit adds a "Share results with the omlx.ai community" checkbox to the throughput benchmark config card, checked by default so existing user behavior is unchanged. The frontend sends the user's choice as should_upload_results on POST /api/bench/start; the run_benchmark coroutine gates the call to _upload_to_omlx_ai on that flag. When the box is unchecked, the SSE 'done' handler finalizes the run directly instead of entering the "Uploading..." UI state. The hint text spells out exactly what gets uploaded and notes the common reason to uncheck (benchmarking a private model whose name would otherwise be included in the payload). The BenchmarkRequest field defaults to True so any direct API caller keeps current behavior. The accuracy benchmark has no upload path and is unaffected.
jabagawee
force-pushed
the
feat/benchmark-upload-opt-out
branch
from
4f450ff to
12ad989
Compare
jabagawee
changed the title
Author
|
updated the PR to have the checkbox checked by default (now it's opt-out instead of opt-in) to match default behavior to existing behavior |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Throughput benchmarks upload results to omlx.ai unconditionally at the end of every run, including a stable per-device hash and the model name (which can be a private/local model id). There is no UI toggle, env var, or request flag to disable it.
This PR adds a "Share results with the omlx.ai community" checkbox to the throughput benchmark config card, checked by default so existing user behavior is unchanged. The frontend sends the user's choice as
should_upload_resultsonPOST /api/bench/start; therun_benchmarkcoroutine gates the call to_upload_to_omlx_aion that flag. When the box is unchecked, the SSEdonehandler finalizes the run directly instead of entering the "Uploading..." UI state.The hint text spells out exactly what gets uploaded and notes the common reason to uncheck (benchmarking a private model whose name would otherwise be included in the payload).
Notes
BenchmarkRequest.should_upload_resultsdefaults toTrue, so any direct API caller keeps current behavior.eni18n strings are added; other locales fall back via the resolver inbase.html.