Description
There is no benches/ anywhere in the workspace and no benchmark dependency, so the engine sits in the request path with no measured cost. Nothing today would catch a change that doubles per-request latency.
Build the suite and take a baseline. What to measure:
- Time. Per-hook overhead and full-decision latency, at p50, p95, and p99. Break out policy evaluation from plugin dispatch, since they scale differently.
- Throughput. Decisions per second under concurrency, including
mode: concurrent in the executor.
- CPU profile. Where the time goes. The evaluator and parser are the candidates; policies compile once, so compile cost matters at load, not per request.
- Memory. Per-decision allocation, and steady-state footprint as policy size and session count grow. Session taint propagates across requests, so session state growth is the one to watch.
Cover the PDPs separately. Cedar, CEL, and OPA have different cost profiles and the choice between them is one an operator makes on performance grounds.
Pick one framework and stay with it. Benchmarks that are never run rot, so decide up front whether these run in CI or on demand, and say so in the Makefile.
Acceptance criteria
- A benchmark suite under
benches/ covering hook overhead, full-decision latency, throughput, and per-PDP cost.
- A
make bench target.
- Baseline numbers recorded in
docs/, with the hardware they were taken on.
- CPU and memory profiles captured for the hot path, with findings written up.
- A decision on whether benchmarks gate CI, recorded in the same doc. If they do, a regression threshold that accounts for runner noise.
Description
There is no
benches/anywhere in the workspace and no benchmark dependency, so the engine sits in the request path with no measured cost. Nothing today would catch a change that doubles per-request latency.Build the suite and take a baseline. What to measure:
mode: concurrentin the executor.Cover the PDPs separately. Cedar, CEL, and OPA have different cost profiles and the choice between them is one an operator makes on performance grounds.
Pick one framework and stay with it. Benchmarks that are never run rot, so decide up front whether these run in CI or on demand, and say so in the Makefile.
Acceptance criteria
benches/covering hook overhead, full-decision latency, throughput, and per-PDP cost.make benchtarget.docs/, with the hardware they were taken on.