feat(eval): real HC3 evaluation — recalibrate Binoculars, correct the accuracy story - #4
Merged
Merged
Conversation
… LLM output Closes the #1 gap from both audits: the detector had never been measured against a large, out-of-distribution corpus. Adds reproducible HC3 evaluation (real human text vs real ChatGPT output) and fixes what it exposed. Added - scripts/prepare_hc3.py: streams the public Hello-SimpleAI/HC3 corpus and builds a balanced JSONL sample. The corpus is NOT committed (third-party data); only the script and the resulting metrics are, so anyone can reproduce. - scripts/calibrate_binoculars.py: fits the Binoculars decision boundary on a stratified calibration half and reports on a HELD-OUT half (no train-on-test). Fixed - Benchmark harness only honoured a calibrated probability named "Ensemble AI Score", so Binoculars was scored through a coarse verdict/confidence step function, distorting ECE and threshold sweeps. It now prefers any analyzer's calibrated primary score. Changed - BinocularsConfig.score_midpoint 0.863 -> 0.7625. The old value was fitted on the bundled 24-sample set and sat INSIDE the real human cluster, flagging 46% of real human text as AI. Refitted on HC3; held-out half: accuracy 1.000, FPR 0.000. Measured on HC3 (n=200, balanced): Binoculars acc 1.000 AUROC 1.000 FPR 0.000 Ensemble acc 0.950 AUROC 0.998 FPR 0.100 GPT-2 alone acc 0.750 AUROC 0.756 FPR 0.500 <- flags half of real human text NLTK alone acc 0.500 AUROC 0.420 FPR 0.000 <- below chance; useful only as a human-side prior in the ensemble Honesty - The bundled data/benchmark "AI" samples are hand-written imitations of LLM style, not real model output, and are not machine-like when measured against real ChatGPT text (ratios 0.72-0.84 vs 0.60-0.76). A correctly-calibrated Binoculars labels most of them human-written -- correctly, since a human wrote them. That set is now documented as a pipeline regression fixture only; its scores are not accuracy. README/docs lead with HC3 numbers instead. Tests: 233 passed, 85% coverage, lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
📝 WalkthroughWalkthroughAdds scripts to download/prepare the HC3 dataset and calibrate the Binoculars decision boundary, updates the default ChangesHC3 Evaluation and Binoculars Recalibration
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant PrepareHC3 as prepare_hc3.py
participant HuggingFace as HC3 corpus
participant Calibrate as calibrate_binoculars.py
participant Analyzer as BinocularsAnalyzer
participant Settings as BinocularsConfig
User->>PrepareHC3: run main(--split, --per-class)
PrepareHC3->>HuggingFace: stream JSONL split
HuggingFace-->>PrepareHC3: records
PrepareHC3-->>User: write HC3 JSONL dataset
User->>Calibrate: run main(--dataset)
Calibrate->>Analyzer: compute_ratios per sample
Analyzer-->>Calibrate: ratio, label rows
Calibrate->>Calibrate: stratified_split, sweep midpoints
Calibrate-->>User: print chosen midpoint
User->>Settings: update score_midpoint default
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Biome (2.5.1)docs/benchmarks/hc3_gpt2_report.jsonFile contains syntax errors that prevent linting: Line 76: String values must be double quoted.; Line 77: String values must be double quoted.; Line 111: String values must be double quoted.; Line 112: String values must be double quoted. docs/benchmarks/hc3_ensemble_report.jsonFile contains syntax errors that prevent linting: Line 111: String values must be double quoted.; Line 112: String values must be double quoted. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
scripts/calibrate_binoculars.py (3)
48-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid calling the private
_compute_binocularsfrom outside the class.The calibration script depends on
BinocularsAnalyzer._compute_binoculars, a leading-underscore method. If the method is renamed or its return signature changes, this script breaks silently with no import-time error. Consider adding a lightweight public accessor (e.g.,compute_ratio(text) -> float) onBinocularsAnalyzerand using that here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/calibrate_binoculars.py` at line 48, The calibration script is calling the private BinocularsAnalyzer._compute_binoculars method directly, which creates a fragile dependency on an internal implementation detail. Add a small public method on BinocularsAnalyzer such as compute_ratio(text) that returns just the ratio, and update the calibration code to use that public accessor instead of _compute_binoculars.
122-122: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrint precision doesn't match the config value.
Line 122 prints
{best_mid:.3f}(3 decimals, e.g.,0.763) butBinocularsConfig.score_midpointis set to0.7625(4 decimals). If someone copies the script output directly, they'd get a slightly different threshold. Consider using:.4ffor consistency.🔧 Suggested fix
- print(f"\nSet BinocularsConfig.score_midpoint = {best_mid:.3f}") + print(f"\nSet BinocularsConfig.score_midpoint = {best_mid:.4f}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/calibrate_binoculars.py` at line 122, The printed BinocularsConfig.score_midpoint value in calibrate_binoculars.py is using 3 decimal places while the actual configured threshold is set with 4-decimal precision. Update the print statement near best_mid so its formatting matches the value assigned to BinocularsConfig.score_midpoint (use the same precision as the config output) to keep the copied threshold consistent.
74-96: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract confusion-matrix helper to eliminate duplication.
balanced_accuracyandreportindependently compute the same TP/FN/TN/FP counts. Extract a small_confusion_matrix(rows, midpoint)helper returning the four counts, then have both functions consume it.♻️ Suggested refactor
def balanced_accuracy(rows: List[Dict], midpoint: float) -> float: """Lower ratio => AI. Balanced accuracy at a candidate midpoint.""" - tp = sum(1 for r in rows if r["label"] == 1 and r["ratio"] < midpoint) - fn = sum(1 for r in rows if r["label"] == 1 and r["ratio"] >= midpoint) - tn = sum(1 for r in rows if r["label"] == 0 and r["ratio"] >= midpoint) - fp = sum(1 for r in rows if r["label"] == 0 and r["ratio"] < midpoint) + tp, fn, tn, fp = _confusion_matrix(rows, midpoint) tpr = tp / (tp + fn) if (tp + fn) else 0.0 tnr = tn / (tn + fp) if (tn + fp) else 0.0 return (tpr + tnr) / 2 +def _confusion_matrix(rows: List[Dict], midpoint: float) -> Tuple[int, int, int, int]: + tp = sum(1 for r in rows if r["label"] == 1 and r["ratio"] < midpoint) + fn = sum(1 for r in rows if r["label"] == 1 and r["ratio"] >= midpoint) + tn = sum(1 for r in rows if r["label"] == 0 and r["ratio"] >= midpoint) + fp = sum(1 for r in rows if r["label"] == 0 and r["ratio"] < midpoint) + return tp, fn, tn, fp + def report(rows: List[Dict], midpoint: float, name: str) -> None: - tp = sum(1 for r in rows if r["label"] == 1 and r["ratio"] < midpoint) - fn = sum(1 for r in rows if r["label"] == 1 and r["ratio"] >= midpoint) - tn = sum(1 for r in rows if r["label"] == 0 and r["ratio"] >= midpoint) - fp = sum(1 for r in rows if r["label"] == 0 and r["ratio"] < midpoint) + tp, fn, tn, fp = _confusion_matrix(rows, midpoint) n = len(rows)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/calibrate_binoculars.py` around lines 74 - 96, Both balanced_accuracy and report duplicate the same TP/FN/TN/FP calculations; extract a shared _confusion_matrix(rows, midpoint) helper in scripts/calibrate_binoculars.py and have both functions call it. Keep the existing behavior the same, but let balanced_accuracy compute tpr/tnr from the helper’s counts and let report format its metrics from the same source of truth.src/evaluation/benchmark.py (2)
49-61: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLogic fix looks correct; add Args/Returns docstring blocks.
The name-based lookup correctly matches the upstream contracts (
AnalysisResult.get_scorereturns the first match by name, and both Binoculars and Ensemble publish calibrated scores under these exact names), fixing the prior bug where only"Ensemble AI Score"was checked. However, the docstring omits the Google-styleArgs/Returnsblocks required for public functions.📝 Proposed docstring addition
Prefers an analyzer's own **calibrated** primary score when it exposes one (the ensemble and Binoculars do). Only analyzers without a calibrated probability fall back to a coarse monotone score derived from the verdict direction and confidence — that fallback is a step function, so it distorts calibration metrics (ECE) and threshold sweeps. + + Args: + result: The analyzer's :class:`AnalysisResult` to convert. + + Returns: + A float AI-probability clamped to ``[0, 1]``. """As per coding guidelines, "Public methods should include Google-style
ArgsandReturnsdocumentation blocks."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/evaluation/benchmark.py` around lines 49 - 61, The logic in result_to_ai_probability is fine, but its docstring needs Google-style Args and Returns sections to satisfy the public API documentation standard. Update the function’s docstring to document the AnalysisResult parameter in an Args block and the float return value in a Returns block, keeping the existing summary of how the calibrated score lookup works.Source: Coding guidelines
40-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the analyzers' score-name constants here
BinocularsAnalyzer.PRIMARY_SCORE_NAMEandEnsembleAnalyzer.ENSEMBLE_SCORE_NAMEare the canonical strings. Prefer them over literals, but keep the imports lazy so this benchmark module doesn't eagerly load the heavy analyzers at import time.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/evaluation/benchmark.py` around lines 40 - 43, Replace the hard-coded score-name literals in benchmark.py with the canonical constants from BinocularsAnalyzer and EnsembleAnalyzer, and keep those imports lazy so the module does not eagerly load the heavy analyzers at import time. Update the _PRIMARY_SCORE_NAMES definition to source the values from BinocularsAnalyzer.PRIMARY_SCORE_NAME and EnsembleAnalyzer.ENSEMBLE_SCORE_NAME, using local/lazy imports or an equivalent deferred lookup near _PRIMARY_SCORE_NAMES so the benchmark module still imports cheaply.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/benchmarks/hc3_ensemble_report.json`:
- Around line 107-113: The calibration report is serializing bare NaN values for
empty bins, which makes the JSON invalid for strict parsers. Update
src/evaluation/metrics.py in calibration_bins() so zero-count bins return
null-friendly values or are omitted instead of float("nan"), then regenerate
hc3_ensemble_report.json to reflect the new serialization.
In `@src/config/settings.py`:
- Around line 142-151: Update the Settings class docstring in settings.py so it
matches the new HC3-based calibration described near score_midpoint. The current
class-level description still mentions the bundled benchmark, which is now
stale; revise that wording to say the decision midpoint was re-fitted on the HC3
corpus and keep the explanation aligned with the score_midpoint documentation.
Locate the class docstring for Settings and adjust only the outdated calibration
sentence.
---
Nitpick comments:
In `@scripts/calibrate_binoculars.py`:
- Line 48: The calibration script is calling the private
BinocularsAnalyzer._compute_binoculars method directly, which creates a fragile
dependency on an internal implementation detail. Add a small public method on
BinocularsAnalyzer such as compute_ratio(text) that returns just the ratio, and
update the calibration code to use that public accessor instead of
_compute_binoculars.
- Line 122: The printed BinocularsConfig.score_midpoint value in
calibrate_binoculars.py is using 3 decimal places while the actual configured
threshold is set with 4-decimal precision. Update the print statement near
best_mid so its formatting matches the value assigned to
BinocularsConfig.score_midpoint (use the same precision as the config output) to
keep the copied threshold consistent.
- Around line 74-96: Both balanced_accuracy and report duplicate the same
TP/FN/TN/FP calculations; extract a shared _confusion_matrix(rows, midpoint)
helper in scripts/calibrate_binoculars.py and have both functions call it. Keep
the existing behavior the same, but let balanced_accuracy compute tpr/tnr from
the helper’s counts and let report format its metrics from the same source of
truth.
In `@src/evaluation/benchmark.py`:
- Around line 49-61: The logic in result_to_ai_probability is fine, but its
docstring needs Google-style Args and Returns sections to satisfy the public API
documentation standard. Update the function’s docstring to document the
AnalysisResult parameter in an Args block and the float return value in a
Returns block, keeping the existing summary of how the calibrated score lookup
works.
- Around line 40-43: Replace the hard-coded score-name literals in benchmark.py
with the canonical constants from BinocularsAnalyzer and EnsembleAnalyzer, and
keep those imports lazy so the module does not eagerly load the heavy analyzers
at import time. Update the _PRIMARY_SCORE_NAMES definition to source the values
from BinocularsAnalyzer.PRIMARY_SCORE_NAME and
EnsembleAnalyzer.ENSEMBLE_SCORE_NAME, using local/lazy imports or an equivalent
deferred lookup near _PRIMARY_SCORE_NAMES so the benchmark module still imports
cheaply.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 06e1e75a-39c6-44b1-acb4-a2f5cae25049
⛔ Files ignored due to path filters (6)
docs/benchmarks/calibration_hc3_binoculars.pngis excluded by!**/*.pngdocs/benchmarks/calibration_hc3_ensemble.pngis excluded by!**/*.pngdocs/benchmarks/calibration_hc3_gpt2.pngis excluded by!**/*.pngdocs/benchmarks/roc_hc3_binoculars.pngis excluded by!**/*.pngdocs/benchmarks/roc_hc3_ensemble.pngis excluded by!**/*.pngdocs/benchmarks/roc_hc3_gpt2.pngis excluded by!**/*.png
📒 Files selected for processing (13)
.gitignoreCHANGELOG.mdREADME.mddata/benchmark/README.mddocs/benchmarks/README.mddocs/benchmarks/hc3_binoculars_report.jsondocs/benchmarks/hc3_ensemble_report.jsondocs/benchmarks/hc3_gpt2_report.jsonscripts/calibrate_binoculars.pyscripts/prepare_hc3.pysrc/config/settings.pysrc/evaluation/benchmark.pytests/test_evaluation.py
Comment on lines
+107
to
+113
| { | ||
| "bin_lower": 0.9, | ||
| "bin_upper": 1.0, | ||
| "count": 0, | ||
| "mean_predicted": NaN, | ||
| "observed_fraction": NaN | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Report excerpt =="
nl -ba docs/benchmarks/hc3_ensemble_report.json | sed -n '100,120p'
echo
echo "== Search for report generation / calibration bins =="
rg -n "calibration_bins|hc3_ensemble_report|mean_predicted|observed_fraction|NaN" -S .
echo
echo "== File metadata =="
stat -c '%n %s bytes' docs/benchmarks/hc3_ensemble_report.jsonRepository: satyamshivam13/AI_Text_Detector
Length of output: 230
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Report excerpt =="
python3 - <<'PY'
from pathlib import Path
p = Path("docs/benchmarks/hc3_ensemble_report.json")
for i, line in enumerate(p.read_text().splitlines(), 1):
if 100 <= i <= 120:
print(f"{i:4d}: {line}")
PY
echo
echo "== Strict JSON parse check =="
python3 - <<'PY'
import json
from pathlib import Path
p = Path("docs/benchmarks/hc3_ensemble_report.json")
text = p.read_text()
try:
json.loads(text)
print("json.loads: OK")
except Exception as e:
print(f"json.loads: {type(e).__name__}: {e}")
PY
echo
echo "== Search for calibration/report generator =="
python3 - <<'PY'
from pathlib import Path
for path in Path(".").rglob("*"):
if path.is_file() and path.suffix in {".py", ".ts", ".js", ".json", ".md"}:
try:
text = path.read_text(errors="ignore")
except Exception:
continue
if "calibration_bins" in text or "hc3_ensemble_report" in text or "mean_predicted" in text or "observed_fraction" in text:
print(path)
PYRepository: satyamshivam13/AI_Text_Detector
Length of output: 1097
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Relevant source outlines =="
python3 - <<'PY'
from pathlib import Path
for p in [Path("src/evaluation/metrics.py"), Path("src/evaluation/benchmark.py")]:
print(f"\n## {p}")
if p.exists():
for i, line in enumerate(p.read_text().splitlines(), 1):
if any(k in line for k in ["def ", "json", "calibration_bins", "NaN", "nan", "dump", "dumps"]):
print(f"{i:4d}: {line}")
else:
print("missing")
PY
echo
echo "== Strict JSON consumer check in Node =="
node - <<'JS'
const fs = require('fs');
const text = fs.readFileSync('docs/benchmarks/hc3_ensemble_report.json', 'utf8');
try {
JSON.parse(text);
console.log('JSON.parse: OK');
} catch (e) {
console.log(e.name + ': ' + e.message);
}
JS
echo
echo "== Find exact NaN emission path =="
python3 - <<'PY'
from pathlib import Path
for path in [Path("src/evaluation/metrics.py"), Path("src/evaluation/benchmark.py")]:
if path.exists():
print(f"\n## {path}")
lines = path.read_text().splitlines()
for start in range(1, len(lines)+1):
line = lines[start-1]
if "calibration" in line or "json.dump" in line or "json.dumps" in line or "nan" in line.lower():
lo = max(1, start-4)
hi = min(len(lines), start+8)
for i in range(lo, hi+1):
print(f"{i:4d}: {lines[i-1]}")
print("---")
PYRepository: satyamshivam13/AI_Text_Detector
Length of output: 12356
Empty calibration bins should not serialize NaN
docs/benchmarks/hc3_ensemble_report.json:107-113 emits bare NaN for the zero-count bin, which breaks strict JSON consumers like JSON.parse and jq. src/evaluation/metrics.py::calibration_bins() should return null or skip empty bins instead of float("nan"), then regenerate the report.
🧰 Tools
🪛 Biome (2.5.1)
[error] 111-111: String values must be double quoted.
(parse)
[error] 112-112: String values must be double quoted.
(parse)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/benchmarks/hc3_ensemble_report.json` around lines 107 - 113, The
calibration report is serializing bare NaN values for empty bins, which makes
the JSON invalid for strict parsers. Update src/evaluation/metrics.py in
calibration_bins() so zero-count bins return null-friendly values or are omitted
instead of float("nan"), then regenerate hc3_ensemble_report.json to reflect the
new serialization.
Comment on lines
+142
to
+151
| # Lower ratio => more AI. Midpoint fitted on the HC3 corpus (real human text | ||
| # vs real ChatGPT output): human ratios ~0.77-1.10, AI ~0.60-0.76. Fitted on a | ||
| # calibration half and validated on a held-out half (accuracy 1.000, FPR | ||
| # 0.000) via scripts/calibrate_binoculars.py. | ||
| # | ||
| # NOTE: the previous value (0.863) was fitted on the tiny bundled benchmark, | ||
| # whose "AI" samples are hand-written imitations of LLM style rather than real | ||
| # model output. It sat inside the real human cluster and flagged 46% of real | ||
| # human text as AI. Re-fit on real LLM output before trusting any new value. | ||
| score_midpoint: float = 0.7625 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Update stale class docstring to reflect HC3 calibration.
Line 135 (class docstring) still says "the decision midpoint is calibrated on the bundled benchmark," but the new comment on lines 142-150 documents that it was re-fitted on the HC3 corpus. This inconsistency could mislead readers.
📝 Suggested fix
A small observer/performer pair sharing one tokenizer keeps it local and
- CPU-friendly; the decision midpoint is calibrated on the bundled benchmark.
+ CPU-friendly; the decision midpoint is calibrated on the HC3 corpus (real
+ human text vs real ChatGPT output). See scripts/calibrate_binoculars.py.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/config/settings.py` around lines 142 - 151, Update the Settings class
docstring in settings.py so it matches the new HC3-based calibration described
near score_midpoint. The current class-level description still mentions the
bundled benchmark, which is now stale; revise that wording to say the decision
midpoint was re-fitted on the HC3 corpus and keep the explanation aligned with
the score_midpoint documentation. Locate the class docstring for Settings and
adjust only the outdated calibration sentence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the #1 gap identified by both audits: this detector had never been measured against a large, out-of-distribution corpus. Now it has been — and the results changed what we can honestly claim.
Measured on HC3 (real human text vs real ChatGPT output, n=200 balanced)
What this exposed
1. The Binoculars boundary was badly miscalibrated. It showed AUROC 1.000 with FPR 0.460 — perfect ranking, useless threshold. The midpoint (0.863) had been fitted on the tiny bundled set and sat inside the real human cluster, flagging 46% of real human text as AI. Refitted on HC3 with a proper held-out split (
scripts/calibrate_binoculars.py): fit on one half, validated on the other → accuracy 1.000, FPR 0.000. New midpoint 0.7625.2. A harness bug hid it.
result_to_ai_probabilityonly honoured a calibrated score named"Ensemble AI Score", so Binoculars was evaluated through a coarse verdict/confidence step function, distorting ECE and threshold sweeps. Fixed + tested.3. Our own bundled benchmark's "AI" class is invalid. Its "AI" samples are hand-written imitations of LLM style, not real model output — ratios 0.72–0.84 vs 0.60–0.76 for real ChatGPT. With the correct boundary, Binoculars labels 11/12 of them human-written — and it is right to, because a human wrote them. That set is now documented as a pipeline regression fixture only; its scores are not accuracy. This invalidates the previously-headlined "FPR 0.000 on the bundled set" claim, and the docs now say so.
4. GPT-2 alone flags half of real human text as AI. Documented prominently — do not use the GPT-2-only app to make decisions about people. NLTK alone is below chance, but earns its keep in the ensemble as a human-side prior that corrects GPT-2's over-flagging (which is why 0.950 > 0.750).
Added
scripts/prepare_hc3.py— streams the public HC3 corpus into our JSONL schema. The corpus is not committed (third-party data); only the script and metrics are, so results are reproducible. Committed reports contain ids/scores, no corpus text.scripts/calibrate_binoculars.py— honest fit/held-out boundary calibration.docs/benchmarks/.Docs
README,
docs/benchmarks/README.md, anddata/benchmark/README.mdall rewritten to lead with HC3 and carry the limitations (ChatGPT-era only; no adversarial/paraphrase/mixed/non-English/ESL-subpopulation evaluation; ensemble ECE 0.201 means trust the ranking, not the confidence number).Validation
233 passed, 85% coverage, lint clean.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Chores