feat(eval): real HC3 evaluation — recalibrate Binoculars, correct the accuracy story

.gitignore

@@ -57,3 +57,6 @@ Thumbs.db
 # Local tooling artifacts (not part of the repo)
 .obsidian/
 graphify-out/
+
+# Third-party evaluation corpora (downloaded, not redistributed)
+data/external/

CHANGELOG.md

@@ -6,6 +6,34 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+- **HC3 evaluation.** `scripts/prepare_hc3.py` builds a balanced sample of the
+  public HC3 corpus (real human text vs real ChatGPT output); the corpus itself is
+  not committed. First out-of-distribution measurement of this detector.
+- `scripts/calibrate_binoculars.py` — fits the Binoculars decision boundary on a
+  calibration half and validates it on a **held-out** half.
+
+### Changed
+- **Binoculars `score_midpoint` recalibrated on real LLM output**: 0.863 → 0.7625.
+  The old value was fitted on the bundled set and sat inside the *real* human
+  cluster, flagging **46% of real human text as AI**. On HC3 the new boundary gives
+  accuracy 1.000 / FPR 0.000, validated on a held-out half.
+- Documentation now leads with HC3 numbers, not the bundled set's.
+
+### Fixed
+- The benchmark harness ignored an analyzer's own calibrated probability unless it
+  was named `"Ensemble AI Score"`, so Binoculars was evaluated through a coarse
+  verdict/confidence step function (distorting ECE and threshold sweeps). It now
+  prefers any calibrated primary score.
+
+### Honesty
+- The bundled `data/benchmark/` "AI" samples are **hand-written imitations of LLM
+  style, not real model output**, and are not machine-like when measured against
+  real ChatGPT text. That set is a pipeline regression fixture only; its scores are
+  **not** accuracy. Documented in `data/benchmark/README.md`.
+- Measured on HC3: **GPT-2 alone flags 50% of real human text as AI**; NLTK alone
+  is below chance (AUROC 0.420).
+
 ## [2.0.0] - 2026-07-08
 
 Remediation of the project audit. Highlights: the ensemble no longer flags

README.md

@@ -53,12 +53,18 @@ This project ships **three independent Streamlit apps** — one per detection en
 
 **Not sure which to use?** Start with `app.py` (NLTK) — it is the lightest and needs no model download.
 
-### Experimental: Binoculars (cross-perplexity)
+### Binoculars (cross-perplexity) — the most accurate analyzer
 
-A modern two-model detector (`gpt2` + `distilgpt2`) after Hans et al., 2024 — far
-more robust than single-model perplexity. Available as a standalone analyzer and
-via the benchmark CLI (`--analyzer binoculars`); on the bundled set it scores
-AUROC 1.000 / FPR 0.000. See [docs/benchmarks/](docs/benchmarks/).
+A two-model detector (`gpt2` + `distilgpt2`) after Hans et al., 2024. It scores
+text by the ratio of an observer model's log-perplexity to the observer/performer
+cross-perplexity, which cancels the prompt/topic bias that makes single-model
+perplexity brittle. **On real human-vs-ChatGPT text (HC3) it scores accuracy
+1.000 with FPR 0.000**, with its decision boundary fitted on a held-out split.
+See [docs/benchmarks/](docs/benchmarks/).
+
+It is available standalone and via the benchmark CLI (`--analyzer binoculars`);
+it is **not** in the default ensemble (it needs a second model). To fuse it, give
+`EnsembleConfig.weight_binoculars` a non-zero weight and rebalance.
 
 ```python
 from src.analyzers.binoculars_analyzer import BinocularsAnalyzer
@@ -116,23 +122,37 @@ print(result.to_dict())
 ## Accuracy and Evaluation
 
 This project ships a real evaluation layer instead of asking you to take accuracy
-on faith. Run it yourself:
+on faith. Reproduce it yourself against **real human text vs real ChatGPT output**
+(the public HC3 corpus):
 
 ```bash
-python -m src.evaluation.benchmark --analyzer ensemble --plots out/
+python scripts/prepare_hc3.py                    # downloads + samples HC3
+python -m src.evaluation.benchmark --analyzer binoculars \
+    --dataset data/external/hc3_sample.jsonl
 ```
 
-On the small bundled benchmark (`data/benchmark/`), the **calibrated ensemble**
-scores Accuracy/F1/AUROC 1.000 with a **false-positive rate of 0.000** (human
-text is not flagged as AI). See [docs/benchmarks/](docs/benchmarks/) for the full
-report and ROC/calibration plots.
-
-> ⚠️ Those numbers are on a small, in-distribution set — a regression/calibration
-> check, **not** an authoritative accuracy claim. Real-world text (edited,
-> paraphrased, mixed, ESL, technical) is much harder. Evaluate on a large public
-> benchmark (RAID, HC3) via `--dataset` before making any external claim. The
-> NLTK-only signal, in particular, is weak (Brown corpus, 1961) and carries a
-> small ensemble weight for that reason.
+**Measured on HC3 (n=200, balanced):**
+
+| Analyzer | Accuracy | AUROC | **FPR** (human flagged AI) |
+|----------|---------:|------:|---------------------------:|
+| **Binoculars** | **1.000** | 1.000 | **0.000** |
+| Ensemble (GPT-2 + NLTK) | 0.950 | 0.998 | 0.100 |
+| GPT-2 alone | 0.750 | 0.756 | **0.500** |
+| NLTK alone | 0.500 | 0.420 | 0.000 |
+
+**Use Binoculars.** GPT-2 alone flags *half of real human text as AI* — single-model
+perplexity is as brittle as the literature says. NLTK alone is below chance, but
+earns its place in the ensemble as a human-side prior that corrects GPT-2's
+over-flagging. Full report, plots, and the held-out calibration procedure:
+[docs/benchmarks/](docs/benchmarks/).
+
+> ⚠️ **Limits of these numbers.** HC3 is ChatGPT-era output; edited, paraphrased,
+> and human/AI-mixed text are harder and unmeasured. 200 samples means wide
+> confidence intervals. There is no evaluation on adversarial "humanizer" attacks,
+> non-English text, or human sub-populations (ESL writers, students) where false
+> positives do real harm. The bundled `data/benchmark/` set is a **pipeline
+> regression fixture only** — its "AI" samples are hand-written imitations, not
+> model output, so its scores are not accuracy.
 
 ## Limitations and Ethics
 

data/benchmark/README.md

@@ -15,12 +15,22 @@ evaluation layer the project previously lacked.
 - `label` is `"human"` or `"ai"` (the positive/AI class is `1`).
 - `source` is a free-text provenance tag.
 
-## Provenance and honesty
+## ⚠️ Provenance and honesty — the "AI" class is synthetic
 
 - **Human** samples are original casual/idiosyncratic prose written for this
   repository (personal notes, reviews, rants, journal entries).
-- **AI** samples are representative of the formal, hedged, list-structured style
-  produced by general-purpose LLMs.
+- **AI** samples are **hand-written imitations** of the formal, hedged,
+  list-structured style produced by general-purpose LLMs. **They are not real
+  model output.**
+
+This matters. Measured against real ChatGPT text, these imitations are not
+machine-like: Binoculars cross-perplexity ratios for real ChatGPT output are
+0.60–0.76, while these hand-written "AI" samples score 0.72–0.84 — overlapping
+the *human* range of real data. A correctly-calibrated detector therefore labels
+most of them human-written, and it is **right** to do so.
+
+**Never cite this set's numbers as detector accuracy.** Use HC3 (see
+`scripts/prepare_hc3.py` and `docs/benchmarks/README.md`) for that.
 
 This set is **intentionally small (24 samples)** and stylistically clean. It is
 designed for:

docs/benchmarks/README.md

@@ -1,79 +1,90 @@
 # Benchmark Results
 
-Reproducible measurements produced by the evaluation harness:
+Two corpora are used, and **they are not equally trustworthy**:
+
+| Corpus | Size | What it is | Use it for |
+|--------|-----:|------------|------------|
+| **HC3** (`data/external/`) | 200 | Real human answers (Reddit/finance/medicine/open-QA) vs **real ChatGPT output** | **Accuracy. This is the number that counts.** |
+| Bundled (`data/benchmark/`) | 24 | Hand-written human prose vs **hand-written imitations of LLM style** | Pipeline regression only — see the warning below |
+
+Reproduce everything:
 
 ```bash
+python scripts/prepare_hc3.py                    # downloads + samples HC3 (not committed)
 python -m src.evaluation.benchmark --analyzer ensemble \
-    --output docs/benchmarks/ensemble_report.json --plots docs/benchmarks
+    --dataset data/external/hc3_sample.jsonl
 ```
 
-All numbers below are on the **bundled 24-sample benchmark**
-(`data/benchmark/samples.jsonl`), which is small and stylistically clean — see
-that file's README for its scope and honest limitations. These are regression
-and calibration numbers, **not** an authoritative accuracy claim on real-world
-(edited, paraphrased, mixed, ESL, technical) text.
-
-## Ensemble (GPT-2 75% + NLTK 25%, calibrated)
-
-| Metric | Value |
-|--------|------:|
-| Accuracy | 1.000 |
-| Precision | 1.000 |
-| Recall | 1.000 |
-| F1 | 1.000 |
-| AUROC | 1.000 |
-| **False-positive rate** (human flagged as AI) | **0.000** |
-| False-negative rate (AI missed) | 0.000 |
-| Expected calibration error | 0.175 |
-
-![ROC](roc_ensemble.png) ![Calibration](calibration_ensemble.png)
-
-## Why this matters: the C2 fix
-
-The previous ensemble mapped every analyzer's perplexity to an AI-score with
-`1 - perplexity / 500`. Human text has a GPT-2 perplexity around 58, which that
-formula turned into `1 - 58/500 = 0.88` — **88% AI for ordinary human writing.**
-That systematic false-positive bias is the audit's critical finding C2.
-
-The calibrated logistic (per-analyzer midpoint = decision boundary; see
-`src/analyzers/calibration.py`) fixes the direction and scale for each analyzer.
-On the benchmark, human text now sits well below the 0.5 boundary and the
-false-positive rate is **0.000**.
-
-## Binoculars (cross-perplexity, modern)
-
-A two-model detector (observer `gpt2` + performer `distilgpt2`) after Hans et
-al., 2024 — the SOTA-aligned modernization from the competitive audit. It scores
-text by the ratio of the observer's log-perplexity to the observer/performer
-cross-perplexity, which cancels the prompt/topic bias that makes single-model
-GPT-2 perplexity brittle.
-
-| Metric | Value |
-|--------|------:|
-| Accuracy | 1.000 |
-| F1 | 1.000 |
-| AUROC | 1.000 |
-| **False-positive rate** | **0.000** |
-| Expected calibration error | 0.066 |
-
-![ROC](roc_binoculars.png) ![Calibration](calibration_binoculars.png)
-
-On the benchmark, human scores cluster ~0.88–1.05 and AI ~0.72–0.84 with a clean
-gap; the decision midpoint (0.863) sits between the clusters. Reproduce with
-`python -m src.evaluation.benchmark --analyzer binoculars`. It is available as a
-standalone analyzer and is not enabled in the default ensemble (to keep the
-default lightweight — it needs a second model).
-
-## Single-analyzer baselines
-
-| Analyzer | AUROC | Notes |
-|----------|------:|-------|
-| NLTK only | ~0.41 | Brown-corpus (1961) n-gram signal is weak/near-inverted for modern text; carries only 25% ensemble weight for this reason. |
-| Ensemble | 1.000 | GPT-2 perplexity is the dominant, strongly-separating signal. |
-
-Run `python -m src.evaluation.benchmark --analyzer nltk` to reproduce the NLTK
-baseline. The gap is exactly why the ensemble weights GPT-2 heavily.
-
-> These results reflect a clean, in-distribution set. Expect materially lower
-> numbers on adversarial or edited text — evaluate on a large public benchmark
-> (RAID, HC3) via `--dataset` before making any external accuracy claim.
+---
+
+## HC3 — real human text vs real ChatGPT output (n=200, balanced)
+
+| Analyzer | Accuracy | AUROC | **FPR** (human flagged AI) | FNR (AI missed) | ECE |
+|----------|---------:|------:|---------------------------:|----------------:|----:|
+| **Binoculars** | **1.000** | **1.000** | **0.000** | 0.000 | 0.138 |
+| Ensemble (GPT-2 + NLTK) | 0.950 | 0.998 | 0.100 | 0.000 | 0.201 |
+| GPT-2 alone | 0.750 | 0.756 | **0.500** | 0.000 | 0.170 |
+| NLTK alone | 0.500 | 0.420 | 0.000 | 1.000 | 0.472 |
+
+![ROC](roc_hc3_binoculars.png) ![Calibration](calibration_hc3_binoculars.png)
+
+**Read this table carefully:**
+
+- **Binoculars is the detector to use.** Its cross-perplexity ratio separates real
+  human text from real ChatGPT output perfectly on this corpus.
+- **GPT-2 alone flags half of real human text as AI** (FPR 0.500). Do not use the
+  GPT-2-only app to make decisions about people. Single-model perplexity is
+  exactly as brittle as the literature says.
+- **NLTK alone is below chance** (AUROC 0.420). It never flags AI. Its value is
+  as a *human-side prior* inside the ensemble, where it corrects GPT-2's
+  over-flagging — which is why the ensemble (0.950) beats GPT-2 alone (0.750).
+- **AUROC ≫ accuracy means the threshold, not the signal, is wrong.** Before
+  recalibration, Binoculars scored AUROC 1.000 with FPR 0.460: perfect ranking,
+  useless boundary.
+
+### How the Binoculars boundary was fitted
+
+`BinocularsConfig.score_midpoint` is fitted on HC3 with a proper held-out split
+(`scripts/calibrate_binoculars.py`): the corpus is split 50/50 stratified, the
+midpoint is swept on the calibration half, and reported on the **held-out half**.
+
+```
+ratio  human: min=0.7668 med=0.8840 max=1.0975
+ratio  ai   : min=0.5953 med=0.6633 max=0.7624
+fitted midpoint = 0.7625
+HELD-OUT half (n=100): accuracy 1.000, FPR 0.000, FNR 0.000
+```
+
+The previous value (0.863) was fitted on the bundled set and **sat inside the
+real human cluster**, flagging 46% of real human text as AI.
+
+---
+
+## ⚠️ The bundled 24-sample set is NOT an accuracy benchmark
+
+Its "AI" samples are **hand-written imitations of LLM style, not real model
+output**. Real ChatGPT text has Binoculars ratios of 0.60–0.76; the imitations
+score 0.72–0.84 — overlapping the *human* range of real data.
+
+Consequence: with the correctly-fitted boundary, Binoculars classifies 11 of the
+12 bundled "AI" samples as human-written (FNR 0.917) — **and it is right to do
+so**, because a human wrote them. AUROC stays 1.000 (the ranking is fine); only
+the labels are wrong about what they represent.
+
+Keep this set for what it is: a fast, deterministic fixture that exercises the
+pipeline end-to-end. Never cite its numbers as accuracy.
+
+---
+
+## Honest limitations
+
+- HC3 is ChatGPT-era output. Newer models, **edited** or **paraphrased** AI text,
+  and human/AI **mixed** documents are all harder and are not measured here.
+- 200 samples gives wide confidence intervals. Scale up with
+  `--per-class` before making strong claims.
+- No evaluation on adversarial ("humanizer") attacks, non-English text, or human
+  sub-populations (ESL writers, students) where false positives cause real harm.
+  **Publishing per-population FPR is the responsible next step.**
+- Ensemble ECE 0.201 means its *probabilities* are poorly calibrated even though
+  its *ranking* is strong (AUROC 0.998). Treat the confidence number with
+  suspicion; trust the ordering.