Symptom
Every Community Benchmark upload from the Throughput Benchmark tab fails. The upload panel renders the full Cloudflare "Just a moment..." challenge HTML interstitial as the error message for each pp{N} row. 0/N uploaded is the consistent result. Behavior is reproducible:
- Across two different Macs (MacBook + Mac Studio) on different networks
- Against the stock oMLX DMG release (no fork modifications)
- For at least the past several weeks (regression of unknown date)
Reproduction (network-level, no oMLX needed)
$ curl -i -X POST https://omlx.ai/api/benchmarks \
-H "Content-Type: application/json" \
-d '{}'
HTTP/2 403
cf-mitigated: challenge
content-type: text/html; charset=UTF-8
content-length: 5290
server: cloudflare
server-timing: chlray;desc="9fa3b5aac93e012e"
...
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>...Headers and body confirm a Cloudflare Managed Challenge (cf-mitigated: challenge). The challenge fires for every POST to /api/benchmarks regardless of User-Agent:
| UA |
Status |
python-requests/2.32.5 (oMLX's actual UA) |
403 |
| Safari macOS UA |
403 |
| Chrome macOS UA |
403 |
GET /api/benchmarks also returns 403, so the endpoint is fully gated rather than just POST-restricted.
Diagnosis
The omlx.ai infrastructure has Cloudflare protection (Bot Fight Mode or Managed Challenge) enabled on /api/benchmarks. Because oMLX's upload client (requests.post in omlx/admin/benchmark.py:_upload_to_omlx_ai) cannot solve a JS challenge, every submission gets rejected before it reaches the API.
This is not an oMLX client bug. The client is well-formed — it sends a single JSON POST with reasonable headers. Cloudflare is just blocking the entire class of non-browser API traffic.
That said, oMLX did have a related client-side bug: the error path piped the entire HTML challenge body into the dashboard's SSE error event, where <span x-text="r.error"> rendered the markup as the error message. Filed as PR #1192 (sanitize the error string so the panel surfaces a one-line actionable message instead of a 5KB markup blob).
Suggested resolution paths (server-side — you'd know best)
In rough order of effort:
- Allow-list
/api/benchmarks in the omlx.ai Cloudflare config — add a WAF skip rule for POST /api/benchmarks with appropriate rate limiting. Lowest-friction option.
- Move the benchmark API to a separate origin/subdomain that isn't behind the challenge layer (e.g.
api.omlx.ai/benchmarks without managed challenge).
- Switch to API-key auth — issue keys to oMLX clients (e.g. tied to the
owner_hash) and configure Cloudflare to bypass challenge when a valid API key header is present.
- Add a token-exchange endpoint that returns a short-lived signed token after the client passes Cloudflare's Turnstile/JS challenge, then submit the actual benchmark with that token. More complex but preserves bot protection.
Happy to test against any of the above from this environment if useful.
Impact
Community benchmark contributions have been silently failing for an extended period — leaderboard volume on omlx.ai is presumably depressed because of this. Users see a broken upload panel on every run, which (without PR #1192) is also unreadable.
Related
Symptom
Every Community Benchmark upload from the Throughput Benchmark tab fails. The upload panel renders the full Cloudflare "Just a moment..." challenge HTML interstitial as the error message for each
pp{N}row.0/N uploadedis the consistent result. Behavior is reproducible:Reproduction (network-level, no oMLX needed)
$ curl -i -X POST https://omlx.ai/api/benchmarks \ -H "Content-Type: application/json" \ -d '{}' HTTP/2 403 cf-mitigated: challenge content-type: text/html; charset=UTF-8 content-length: 5290 server: cloudflare server-timing: chlray;desc="9fa3b5aac93e012e" ... <!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>...Headers and body confirm a Cloudflare Managed Challenge (
cf-mitigated: challenge). The challenge fires for everyPOSTto/api/benchmarksregardless ofUser-Agent:python-requests/2.32.5(oMLX's actual UA)GET /api/benchmarksalso returns 403, so the endpoint is fully gated rather than just POST-restricted.Diagnosis
The omlx.ai infrastructure has Cloudflare protection (Bot Fight Mode or Managed Challenge) enabled on
/api/benchmarks. Because oMLX's upload client (requests.postinomlx/admin/benchmark.py:_upload_to_omlx_ai) cannot solve a JS challenge, every submission gets rejected before it reaches the API.This is not an oMLX client bug. The client is well-formed — it sends a single JSON
POSTwith reasonable headers. Cloudflare is just blocking the entire class of non-browser API traffic.That said, oMLX did have a related client-side bug: the error path piped the entire HTML challenge body into the dashboard's SSE error event, where
<span x-text="r.error">rendered the markup as the error message. Filed as PR #1192 (sanitize the error string so the panel surfaces a one-line actionable message instead of a 5KB markup blob).Suggested resolution paths (server-side — you'd know best)
In rough order of effort:
/api/benchmarksin the omlx.ai Cloudflare config — add a WAF skip rule forPOST /api/benchmarkswith appropriate rate limiting. Lowest-friction option.api.omlx.ai/benchmarkswithout managed challenge).owner_hash) and configure Cloudflare to bypass challenge when a valid API key header is present.Happy to test against any of the above from this environment if useful.
Impact
Community benchmark contributions have been silently failing for an extended period — leaderboard volume on omlx.ai is presumably depressed because of this. Users see a broken upload panel on every run, which (without PR #1192) is also unreadable.
Related