Oddish

Run evals on Harbor tasks in the cloud.

Oddish extends Harbor with:

• Provider-aware queuing and automatic retries for LLM providers
• Real-time monitoring via dashboard or CLI
• Postgres-backed state and S3 storage for logs

Just replace harbor run with oddish run.

Quick Start

1. Install

uv pip install oddish

Install latest development version

uv pip install "oddish @ git+https://github.com/abundant-ai/oddish.git#subdirectory=oddish"

2. Generate an API key here

• API key generation is restricted during the beta. To request access, contact the maintainer.

export ODDISH_API_KEY="ok_..."

3. Submit a job

# Run a single agent
oddish run -d terminal-bench@2.0 -a codex -m gpt-5.5 --n-trials 3

# Or sweep multiple agents
oddish run -d terminal-bench@2.0 -c job.yaml

Example job.yaml

4. Monitor Progress

oddish status

Documentation

• CLI docs
• CLI package quick start
• Web dashboard
• Cloud backend
• Self-hosting
• Agents

License

Apache License 2.0