Skip to content

Add streaming speed benchmarks for all providers #4

Description

@xmaks82

Description

Create a benchmark script that measures tokens/second for each provider.

Implementation

  • Create benchmarks/speed_test.py
  • Send the same prompt to each configured provider
  • Measure time-to-first-token and tokens/second
  • Output a comparison table
  • Use a standard prompt (e.g., "Explain quantum computing in 200 words")

Expected output

Provider       Model              TTFT    Tokens/s
─────────────────────────────────────────────────
SambaNova      samba-llama-70b    0.3s    580 t/s
Groq           llama-3.3-70b      0.2s    320 t/s
Cerebras       llama3.1-8b        0.1s    450 t/s
...

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions