PromptIntel

AI Prompt Monitoring & Regression Platform

The same prompt, a different answer next week — and nobody told you.

Periodically runs your prompts across OpenAI, Anthropic, and Groq, stores every historical response, diffs each run against the last one to catch silent regressions, and exposes plain backend analytics. No auth, no vanity charts.

---

Screenshots

| | |

---

Why

LLM APIs change underneath you — same prompt, same model, different answer next week: different tone, length, latency, or cost. Nothing tells you until a user complains. PromptIntel runs your prompts on a schedule (or on demand), stores every response, and flags exactly what changed.

Features

• Multi-provider execution — OpenAI, Anthropic, Groq behind one BaseProvider interface
• Scheduled + manual runs — Celery Beat every N hours, plus an on-demand "Run Now"
• Regression detection — difflib similarity, added/removed text, latency/token deltas, derived improved/stable/degraded/error status
• Resilient execution — per-provider timeout + single retry + continue-on-failure
• Plain-JSON analytics — /dashboard returns real aggregates, no chart library
• Full history — every execution and response kept, never overwritten

Tech stack

Layer	Choice
Backend	FastAPI, async SQLAlchemy 2.0 + asyncpg, Alembic, Pydantic v2
Jobs	Celery + Redis (Beat scheduling)
Database	PostgreSQL
Providers	OpenAI, Anthropic, Groq SDKs
Frontend	React + TypeScript + Vite + Tailwind + Axios
Testing	pytest + pytest-asyncio (32 tests, providers mocked)
Infra	Docker Compose locally · Render + Neon + Upstash + Vercel to deploy

API

Method	Endpoint	Description
POST	/projects	Create a project
GET	/projects, /projects/{id}	List / get a project
DELETE	/projects/{id}	Delete a project (cascades)
POST	/prompts	Add a prompt
GET	/prompts/{project_id}	List a project's prompts
PUT / DELETE	/prompts/{id}	Update / delete a prompt
GET	/providers	List providers
PATCH	/providers/{id}/toggle	Enable/disable a provider
POST	/run/{project_id}	Run all prompts now
GET	/executions/{id}, /history/{project_id}	Execution detail / history
GET	/comparison/{project_id}	Regression comparisons
GET	/dashboard, /health	Analytics / health check

Full interactive docs at /docs once the backend is running.

Run locally

Backend

cd backend
cp .env.example .env   # add OPENAI_API_KEY / ANTHROPIC_API_KEY / GROQ_API_KEY
cd ..
docker compose up -d
docker compose exec api alembic upgrade head

API at http://localhost:8000 (docs at /docs). Default providers auto-seed on first request.

Frontend

cd frontend
npm install
npm run dev

Opens at http://localhost:5173.

Tests

docker compose exec api pytest -v

32 tests — CRUD, execution engine, comparison engine, provider factory, analytics — all mocked, no real API calls.

Structure

promptintel/
  backend/app/
    api/            # routes
    services/
      providers/     # BaseProvider + OpenAI/Anthropic/Groq
      execution.py   # retry/timeout/continue-on-error engine
      comparison.py  # difflib regression detection
      analytics.py   # dashboard aggregates
    scheduler/       # Celery + beat
    models/ schemas/ # SQLAlchemy + Pydantic
  backend/alembic/    # migrations
  backend/tests/      # pytest suite
  frontend/src/
    pages/           # Dashboard, Projects, ProjectDetail
    components/ types/
  docker-compose.yml

License

MIT