From 46f37914d445817b1d5df8f0a9ec48e3899275dc Mon Sep 17 00:00:00 2001 From: Artur Date: Sun, 7 Jun 2026 04:42:58 +0300 Subject: [PATCH] docs: add evaluator design checklist --- EVALUATION_GUIDE.md | 4 ++ README.md | 2 + docs/evaluator_design.md | 116 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 122 insertions(+) create mode 100644 docs/evaluator_design.md diff --git a/EVALUATION_GUIDE.md b/EVALUATION_GUIDE.md index 93f5e73..b7c8516 100644 --- a/EVALUATION_GUIDE.md +++ b/EVALUATION_GUIDE.md @@ -9,6 +9,10 @@ After each generation, the orchestrator automatically runs evaluation: 3. Your `evaluate.py` finds the submission file, evaluates it, and saves `results.json` 4. Orchestrator loads `results.json` and adds metrics to feedback prompt +Because the feedback loop optimizes this signal, evaluator design matters. See +[docs/evaluator_design.md](docs/evaluator_design.md) for a checklist on robust +metrics, private data separation, and anti-gaming checks. + ## Location Place `evaluate.py` in: `tasks//data/public/evaluate.py` diff --git a/README.md b/README.md index fa0b445..4eeec7d 100644 --- a/README.md +++ b/README.md @@ -255,6 +255,7 @@ python my-task/data/public/evaluate.py --gen-dir runs/run_1/gen_1 # should wri ``` Full contract, return-format rules, and a complete example: [EVALUATION_GUIDE.md](EVALUATION_GUIDE.md). +For metric robustness and anti-gaming checks, see [docs/evaluator_design.md](docs/evaluator_design.md). --- @@ -264,6 +265,7 @@ Full contract, return-format rules, and a complete example: [EVALUATION_GUIDE.md - [docs/walkthrough.md](docs/walkthrough.md) — detailed custom-task walkthrough - [docs/configuration.md](docs/configuration.md) — agent impls, models, API keys, CLI reference - [EVALUATION_GUIDE.md](EVALUATION_GUIDE.md) — writing `evaluate.py` for a custom task +- [docs/evaluator_design.md](docs/evaluator_design.md) — evaluator robustness, private data separation, anti-gaming checks - [docs/troubleshooting.md](docs/troubleshooting.md) — common errors and fixes ## Citation diff --git a/docs/evaluator_design.md b/docs/evaluator_design.md new file mode 100644 index 0000000..800eeaa --- /dev/null +++ b/docs/evaluator_design.md @@ -0,0 +1,116 @@ +# Evaluator design checklist + +SIA optimizes whatever signal your evaluator writes to `results.json`. Treat the +evaluator as the task specification: if it rewards the wrong behavior, the +self-improvement loop will reliably discover and amplify that behavior. + +Use this checklist when creating or reviewing `data/public/evaluate.py` for a +custom task. + +## 1. Keep the public task and private score separate + +- Put inputs the target agent may read in `data/public/`. +- Put answer keys, hidden tests, and held-out examples in `data/private/`. +- Do not copy private labels, expected outputs, or hidden-case identifiers into + `task.md`, sample submissions, prompts, reference agents, or logs. +- If the target agent runs with direct host filesystem access, remember that + `data/private/` is a convention enforced by the harness, not a security + boundary. Use Docker/external sandboxing when you need hard isolation. + +## 2. Make `results.json` the canonical artifact + +The orchestrator calls: + +```bash +python data/public/evaluate.py --gen-dir runs/run_1/gen_1 +``` + +Your script should find the generation's submission, score it, and write: + +```text +runs/run_1/gen_1/results.json +``` + +Recommended top-level fields: + +```json +{ + "accuracy": 0.82, + "accuracy_percent": 82.0, + "correct": 82, + "incorrect": 18, + "missing": 0, + "invalid": 0, + "total_questions": 100, + "details": [] +} +``` + +Top-level scalar metrics are easiest for the feedback prompt and web dashboard to +summarize. Keep verbose per-example data under `details` so it can be inspected +without hiding the primary score. + +## 3. Score the behavior you actually want + +Weak metrics become optimization targets. Avoid metrics that can be satisfied by +formatting or surface-level hacks unless that is truly the task. + +Examples of fragile signals: + +- keyword counts instead of semantic correctness +- citation counts without checking whether citations support the claim +- exact-string matching when many paraphrases should be valid +- public-test-only scores when the agent can tune directly to those cases +- rewarding verbosity, code size, or tool usage instead of final quality + +Prefer task-grounded checks: + +- compare against hidden labels, unit tests, or reference outputs +- include counterexamples and false-premise cases +- validate required output schemas before scoring +- penalize missing, invalid, duplicate, or unparsable submissions explicitly +- report both aggregate scores and enough details to debug failures + +## 4. Include robustness slices + +A single scalar can hide brittle progress. Add slice metrics that reveal whether +an improvement generalizes: + +- easy vs hard examples +- common vs rare classes +- public/dev vs private/held-out split +- in-distribution vs paraphrased or perturbed inputs +- previously failed examples vs newly introduced hidden cases + +Keep the primary metric stable across generations, but expose slices so the +feedback agent can distinguish real improvement from overfitting. + +## 5. Be deterministic and reproducible + +- Sort inputs before scoring when file order should not matter. +- Use fixed seeds for randomized evaluators. +- Include exact counts, not just percentages. +- Fail loudly for malformed ground truth; count malformed submissions as + `invalid` instead of crashing when possible. +- Make the evaluator runnable locally without a full SIA run. + +A useful smoke test is: + +```bash +python data/public/evaluate.py --gen-dir /tmp/example-gen +python -m json.tool /tmp/example-gen/results.json +``` + +## 6. Review the evaluator as adversarial surface + +Before launching a long run, ask: + +- Could an agent get a high score without solving the task? +- Could it infer private answers from filenames, logs, examples, or prompts? +- Does the evaluator accidentally read the agent's old `results.json` as a new + submission? +- Are hidden examples diverse enough to catch prompt- or format-specific hacks? +- Would a human trust the metric if the generated solution were deployed? + +If the answer is unclear, add a small hidden regression case or an explicit +validation rule before running the self-improvement loop.