diff --git a/.gitignore b/.gitignore index acd928d..62a76eb 100644 --- a/.gitignore +++ b/.gitignore @@ -57,3 +57,6 @@ Thumbs.db # Local tooling artifacts (not part of the repo) .obsidian/ graphify-out/ + +# Third-party evaluation corpora (downloaded, not redistributed) +data/external/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 350752f..2a476e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,34 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added +- **HC3 evaluation.** `scripts/prepare_hc3.py` builds a balanced sample of the + public HC3 corpus (real human text vs real ChatGPT output); the corpus itself is + not committed. First out-of-distribution measurement of this detector. +- `scripts/calibrate_binoculars.py` — fits the Binoculars decision boundary on a + calibration half and validates it on a **held-out** half. + +### Changed +- **Binoculars `score_midpoint` recalibrated on real LLM output**: 0.863 → 0.7625. + The old value was fitted on the bundled set and sat inside the *real* human + cluster, flagging **46% of real human text as AI**. On HC3 the new boundary gives + accuracy 1.000 / FPR 0.000, validated on a held-out half. +- Documentation now leads with HC3 numbers, not the bundled set's. + +### Fixed +- The benchmark harness ignored an analyzer's own calibrated probability unless it + was named `"Ensemble AI Score"`, so Binoculars was evaluated through a coarse + verdict/confidence step function (distorting ECE and threshold sweeps). It now + prefers any calibrated primary score. + +### Honesty +- The bundled `data/benchmark/` "AI" samples are **hand-written imitations of LLM + style, not real model output**, and are not machine-like when measured against + real ChatGPT text. That set is a pipeline regression fixture only; its scores are + **not** accuracy. Documented in `data/benchmark/README.md`. +- Measured on HC3: **GPT-2 alone flags 50% of real human text as AI**; NLTK alone + is below chance (AUROC 0.420). + ## [2.0.0] - 2026-07-08 Remediation of the project audit. Highlights: the ensemble no longer flags diff --git a/README.md b/README.md index af2fc3f..27cb17d 100644 --- a/README.md +++ b/README.md @@ -53,12 +53,18 @@ This project ships **three independent Streamlit apps** — one per detection en **Not sure which to use?** Start with `app.py` (NLTK) — it is the lightest and needs no model download. -### Experimental: Binoculars (cross-perplexity) +### Binoculars (cross-perplexity) — the most accurate analyzer -A modern two-model detector (`gpt2` + `distilgpt2`) after Hans et al., 2024 — far -more robust than single-model perplexity. Available as a standalone analyzer and -via the benchmark CLI (`--analyzer binoculars`); on the bundled set it scores -AUROC 1.000 / FPR 0.000. See [docs/benchmarks/](docs/benchmarks/). +A two-model detector (`gpt2` + `distilgpt2`) after Hans et al., 2024. It scores +text by the ratio of an observer model's log-perplexity to the observer/performer +cross-perplexity, which cancels the prompt/topic bias that makes single-model +perplexity brittle. **On real human-vs-ChatGPT text (HC3) it scores accuracy +1.000 with FPR 0.000**, with its decision boundary fitted on a held-out split. +See [docs/benchmarks/](docs/benchmarks/). + +It is available standalone and via the benchmark CLI (`--analyzer binoculars`); +it is **not** in the default ensemble (it needs a second model). To fuse it, give +`EnsembleConfig.weight_binoculars` a non-zero weight and rebalance. ```python from src.analyzers.binoculars_analyzer import BinocularsAnalyzer @@ -116,23 +122,37 @@ print(result.to_dict()) ## Accuracy and Evaluation This project ships a real evaluation layer instead of asking you to take accuracy -on faith. Run it yourself: +on faith. Reproduce it yourself against **real human text vs real ChatGPT output** +(the public HC3 corpus): ```bash -python -m src.evaluation.benchmark --analyzer ensemble --plots out/ +python scripts/prepare_hc3.py # downloads + samples HC3 +python -m src.evaluation.benchmark --analyzer binoculars \ + --dataset data/external/hc3_sample.jsonl ``` -On the small bundled benchmark (`data/benchmark/`), the **calibrated ensemble** -scores Accuracy/F1/AUROC 1.000 with a **false-positive rate of 0.000** (human -text is not flagged as AI). See [docs/benchmarks/](docs/benchmarks/) for the full -report and ROC/calibration plots. - -> ⚠️ Those numbers are on a small, in-distribution set — a regression/calibration -> check, **not** an authoritative accuracy claim. Real-world text (edited, -> paraphrased, mixed, ESL, technical) is much harder. Evaluate on a large public -> benchmark (RAID, HC3) via `--dataset` before making any external claim. The -> NLTK-only signal, in particular, is weak (Brown corpus, 1961) and carries a -> small ensemble weight for that reason. +**Measured on HC3 (n=200, balanced):** + +| Analyzer | Accuracy | AUROC | **FPR** (human flagged AI) | +|----------|---------:|------:|---------------------------:| +| **Binoculars** | **1.000** | 1.000 | **0.000** | +| Ensemble (GPT-2 + NLTK) | 0.950 | 0.998 | 0.100 | +| GPT-2 alone | 0.750 | 0.756 | **0.500** | +| NLTK alone | 0.500 | 0.420 | 0.000 | + +**Use Binoculars.** GPT-2 alone flags *half of real human text as AI* — single-model +perplexity is as brittle as the literature says. NLTK alone is below chance, but +earns its place in the ensemble as a human-side prior that corrects GPT-2's +over-flagging. Full report, plots, and the held-out calibration procedure: +[docs/benchmarks/](docs/benchmarks/). + +> ⚠️ **Limits of these numbers.** HC3 is ChatGPT-era output; edited, paraphrased, +> and human/AI-mixed text are harder and unmeasured. 200 samples means wide +> confidence intervals. There is no evaluation on adversarial "humanizer" attacks, +> non-English text, or human sub-populations (ESL writers, students) where false +> positives do real harm. The bundled `data/benchmark/` set is a **pipeline +> regression fixture only** — its "AI" samples are hand-written imitations, not +> model output, so its scores are not accuracy. ## Limitations and Ethics diff --git a/data/benchmark/README.md b/data/benchmark/README.md index 5fedd08..bae1872 100644 --- a/data/benchmark/README.md +++ b/data/benchmark/README.md @@ -15,12 +15,22 @@ evaluation layer the project previously lacked. - `label` is `"human"` or `"ai"` (the positive/AI class is `1`). - `source` is a free-text provenance tag. -## Provenance and honesty +## ⚠️ Provenance and honesty — the "AI" class is synthetic - **Human** samples are original casual/idiosyncratic prose written for this repository (personal notes, reviews, rants, journal entries). -- **AI** samples are representative of the formal, hedged, list-structured style - produced by general-purpose LLMs. +- **AI** samples are **hand-written imitations** of the formal, hedged, + list-structured style produced by general-purpose LLMs. **They are not real + model output.** + +This matters. Measured against real ChatGPT text, these imitations are not +machine-like: Binoculars cross-perplexity ratios for real ChatGPT output are +0.60–0.76, while these hand-written "AI" samples score 0.72–0.84 — overlapping +the *human* range of real data. A correctly-calibrated detector therefore labels +most of them human-written, and it is **right** to do so. + +**Never cite this set's numbers as detector accuracy.** Use HC3 (see +`scripts/prepare_hc3.py` and `docs/benchmarks/README.md`) for that. This set is **intentionally small (24 samples)** and stylistically clean. It is designed for: diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index cc344ce..2ae07ba 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -1,79 +1,90 @@ # Benchmark Results -Reproducible measurements produced by the evaluation harness: +Two corpora are used, and **they are not equally trustworthy**: + +| Corpus | Size | What it is | Use it for | +|--------|-----:|------------|------------| +| **HC3** (`data/external/`) | 200 | Real human answers (Reddit/finance/medicine/open-QA) vs **real ChatGPT output** | **Accuracy. This is the number that counts.** | +| Bundled (`data/benchmark/`) | 24 | Hand-written human prose vs **hand-written imitations of LLM style** | Pipeline regression only — see the warning below | + +Reproduce everything: ```bash +python scripts/prepare_hc3.py # downloads + samples HC3 (not committed) python -m src.evaluation.benchmark --analyzer ensemble \ - --output docs/benchmarks/ensemble_report.json --plots docs/benchmarks + --dataset data/external/hc3_sample.jsonl ``` -All numbers below are on the **bundled 24-sample benchmark** -(`data/benchmark/samples.jsonl`), which is small and stylistically clean — see -that file's README for its scope and honest limitations. These are regression -and calibration numbers, **not** an authoritative accuracy claim on real-world -(edited, paraphrased, mixed, ESL, technical) text. - -## Ensemble (GPT-2 75% + NLTK 25%, calibrated) - -| Metric | Value | -|--------|------:| -| Accuracy | 1.000 | -| Precision | 1.000 | -| Recall | 1.000 | -| F1 | 1.000 | -| AUROC | 1.000 | -| **False-positive rate** (human flagged as AI) | **0.000** | -| False-negative rate (AI missed) | 0.000 | -| Expected calibration error | 0.175 | - -![ROC](roc_ensemble.png) ![Calibration](calibration_ensemble.png) - -## Why this matters: the C2 fix - -The previous ensemble mapped every analyzer's perplexity to an AI-score with -`1 - perplexity / 500`. Human text has a GPT-2 perplexity around 58, which that -formula turned into `1 - 58/500 = 0.88` — **88% AI for ordinary human writing.** -That systematic false-positive bias is the audit's critical finding C2. - -The calibrated logistic (per-analyzer midpoint = decision boundary; see -`src/analyzers/calibration.py`) fixes the direction and scale for each analyzer. -On the benchmark, human text now sits well below the 0.5 boundary and the -false-positive rate is **0.000**. - -## Binoculars (cross-perplexity, modern) - -A two-model detector (observer `gpt2` + performer `distilgpt2`) after Hans et -al., 2024 — the SOTA-aligned modernization from the competitive audit. It scores -text by the ratio of the observer's log-perplexity to the observer/performer -cross-perplexity, which cancels the prompt/topic bias that makes single-model -GPT-2 perplexity brittle. - -| Metric | Value | -|--------|------:| -| Accuracy | 1.000 | -| F1 | 1.000 | -| AUROC | 1.000 | -| **False-positive rate** | **0.000** | -| Expected calibration error | 0.066 | - -![ROC](roc_binoculars.png) ![Calibration](calibration_binoculars.png) - -On the benchmark, human scores cluster ~0.88–1.05 and AI ~0.72–0.84 with a clean -gap; the decision midpoint (0.863) sits between the clusters. Reproduce with -`python -m src.evaluation.benchmark --analyzer binoculars`. It is available as a -standalone analyzer and is not enabled in the default ensemble (to keep the -default lightweight — it needs a second model). - -## Single-analyzer baselines - -| Analyzer | AUROC | Notes | -|----------|------:|-------| -| NLTK only | ~0.41 | Brown-corpus (1961) n-gram signal is weak/near-inverted for modern text; carries only 25% ensemble weight for this reason. | -| Ensemble | 1.000 | GPT-2 perplexity is the dominant, strongly-separating signal. | - -Run `python -m src.evaluation.benchmark --analyzer nltk` to reproduce the NLTK -baseline. The gap is exactly why the ensemble weights GPT-2 heavily. - -> These results reflect a clean, in-distribution set. Expect materially lower -> numbers on adversarial or edited text — evaluate on a large public benchmark -> (RAID, HC3) via `--dataset` before making any external accuracy claim. +--- + +## HC3 — real human text vs real ChatGPT output (n=200, balanced) + +| Analyzer | Accuracy | AUROC | **FPR** (human flagged AI) | FNR (AI missed) | ECE | +|----------|---------:|------:|---------------------------:|----------------:|----:| +| **Binoculars** | **1.000** | **1.000** | **0.000** | 0.000 | 0.138 | +| Ensemble (GPT-2 + NLTK) | 0.950 | 0.998 | 0.100 | 0.000 | 0.201 | +| GPT-2 alone | 0.750 | 0.756 | **0.500** | 0.000 | 0.170 | +| NLTK alone | 0.500 | 0.420 | 0.000 | 1.000 | 0.472 | + +![ROC](roc_hc3_binoculars.png) ![Calibration](calibration_hc3_binoculars.png) + +**Read this table carefully:** + +- **Binoculars is the detector to use.** Its cross-perplexity ratio separates real + human text from real ChatGPT output perfectly on this corpus. +- **GPT-2 alone flags half of real human text as AI** (FPR 0.500). Do not use the + GPT-2-only app to make decisions about people. Single-model perplexity is + exactly as brittle as the literature says. +- **NLTK alone is below chance** (AUROC 0.420). It never flags AI. Its value is + as a *human-side prior* inside the ensemble, where it corrects GPT-2's + over-flagging — which is why the ensemble (0.950) beats GPT-2 alone (0.750). +- **AUROC ≫ accuracy means the threshold, not the signal, is wrong.** Before + recalibration, Binoculars scored AUROC 1.000 with FPR 0.460: perfect ranking, + useless boundary. + +### How the Binoculars boundary was fitted + +`BinocularsConfig.score_midpoint` is fitted on HC3 with a proper held-out split +(`scripts/calibrate_binoculars.py`): the corpus is split 50/50 stratified, the +midpoint is swept on the calibration half, and reported on the **held-out half**. + +``` +ratio human: min=0.7668 med=0.8840 max=1.0975 +ratio ai : min=0.5953 med=0.6633 max=0.7624 +fitted midpoint = 0.7625 +HELD-OUT half (n=100): accuracy 1.000, FPR 0.000, FNR 0.000 +``` + +The previous value (0.863) was fitted on the bundled set and **sat inside the +real human cluster**, flagging 46% of real human text as AI. + +--- + +## ⚠️ The bundled 24-sample set is NOT an accuracy benchmark + +Its "AI" samples are **hand-written imitations of LLM style, not real model +output**. Real ChatGPT text has Binoculars ratios of 0.60–0.76; the imitations +score 0.72–0.84 — overlapping the *human* range of real data. + +Consequence: with the correctly-fitted boundary, Binoculars classifies 11 of the +12 bundled "AI" samples as human-written (FNR 0.917) — **and it is right to do +so**, because a human wrote them. AUROC stays 1.000 (the ranking is fine); only +the labels are wrong about what they represent. + +Keep this set for what it is: a fast, deterministic fixture that exercises the +pipeline end-to-end. Never cite its numbers as accuracy. + +--- + +## Honest limitations + +- HC3 is ChatGPT-era output. Newer models, **edited** or **paraphrased** AI text, + and human/AI **mixed** documents are all harder and are not measured here. +- 200 samples gives wide confidence intervals. Scale up with + `--per-class` before making strong claims. +- No evaluation on adversarial ("humanizer") attacks, non-English text, or human + sub-populations (ESL writers, students) where false positives cause real harm. + **Publishing per-population FPR is the responsible next step.** +- Ensemble ECE 0.201 means its *probabilities* are poorly calibrated even though + its *ranking* is strong (AUROC 0.998). Treat the confidence number with + suspicion; trust the ordering. diff --git a/docs/benchmarks/calibration_hc3_binoculars.png b/docs/benchmarks/calibration_hc3_binoculars.png new file mode 100644 index 0000000..251cd03 Binary files /dev/null and b/docs/benchmarks/calibration_hc3_binoculars.png differ diff --git a/docs/benchmarks/calibration_hc3_ensemble.png b/docs/benchmarks/calibration_hc3_ensemble.png new file mode 100644 index 0000000..661acc5 Binary files /dev/null and b/docs/benchmarks/calibration_hc3_ensemble.png differ diff --git a/docs/benchmarks/calibration_hc3_gpt2.png b/docs/benchmarks/calibration_hc3_gpt2.png new file mode 100644 index 0000000..3d9a438 Binary files /dev/null and b/docs/benchmarks/calibration_hc3_gpt2.png differ diff --git a/docs/benchmarks/hc3_binoculars_report.json b/docs/benchmarks/hc3_binoculars_report.json new file mode 100644 index 0000000..c7715ca --- /dev/null +++ b/docs/benchmarks/hc3_binoculars_report.json @@ -0,0 +1,1717 @@ +{ + "analyzer": "binoculars", + "n_samples": 200, + "metrics_at_0.5": { + "threshold": 0.5, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 100, + "false_positives": 0, + "true_negatives": 100, + "false_negatives": 0, + "accuracy": 1.0, + "precision": 1.0, + "recall": 1.0, + "f1": 1.0, + "specificity": 1.0, + "false_positive_rate": 0.0, + "false_negative_rate": 0.0, + "roc_auc": 1.0, + "expected_calibration_error": 0.13788633850815624 + }, + "best_f1_threshold": 0.5005, + "metrics_at_best_f1": { + "threshold": 0.5005257729838309, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 100, + "false_positives": 0, + "true_negatives": 100, + "false_negatives": 0, + "accuracy": 1.0, + "precision": 1.0, + "recall": 1.0, + "f1": 1.0, + "specificity": 1.0, + "false_positive_rate": 0.0, + "false_negative_rate": 0.0, + "roc_auc": 1.0, + "expected_calibration_error": 0.13788633850815624 + }, + "calibration_bins": [ + { + "bin_lower": 0.0, + "bin_upper": 0.1, + "count": 56, + "mean_predicted": 0.04315613897863579, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.1, + "bin_upper": 0.2, + "count": 23, + "mean_predicted": 0.1378182220283235, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.2, + "bin_upper": 0.30000000000000004, + "count": 13, + "mean_predicted": 0.2412686522450163, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.30000000000000004, + "bin_upper": 0.4, + "count": 4, + "mean_predicted": 0.36667071947052426, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.4, + "bin_upper": 0.5, + "count": 4, + "mean_predicted": 0.44143061891203417, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.5, + "bin_upper": 0.6000000000000001, + "count": 3, + "mean_predicted": 0.5446708269240236, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.6000000000000001, + "bin_upper": 0.7000000000000001, + "count": 6, + "mean_predicted": 0.6436717568590423, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.7000000000000001, + "bin_upper": 0.8, + "count": 14, + "mean_predicted": 0.7453686208562996, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.8, + "bin_upper": 0.9, + "count": 46, + "mean_predicted": 0.8625268046090969, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.9, + "bin_upper": 1.0, + "count": 31, + "mean_predicted": 0.9280889127292343, + "observed_fraction": 1.0 + } + ], + "predictions": [ + { + "id": "hc3-human-0055", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0534, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0012", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2223, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0076", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1437, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0076", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9224, + "verdict": "AI-Generated", + "confidence": 92.7 + }, + { + "id": "hc3-ai-0063", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5005, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0071", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.89, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-human-0015", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1221, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0029", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0996, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0040", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0224, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0039", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8233, + "verdict": "AI-Generated", + "confidence": 89.7 + }, + { + "id": "hc3-human-0001", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0292, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0041", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.275, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0094", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8637, + "verdict": "AI-Generated", + "confidence": 90.9 + }, + { + "id": "hc3-ai-0057", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8945, + "verdict": "AI-Generated", + "confidence": 91.8 + }, + { + "id": "hc3-ai-0026", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.922, + "verdict": "AI-Generated", + "confidence": 92.7 + }, + { + "id": "hc3-human-0048", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0248, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0022", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0117, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0083", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0012, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0004", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8717, + "verdict": "AI-Generated", + "confidence": 91.2 + }, + { + "id": "hc3-human-0042", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0565, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0090", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7466, + "verdict": "Likely AI-Generated", + "confidence": 74.9 + }, + { + "id": "hc3-ai-0046", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8614, + "verdict": "AI-Generated", + "confidence": 90.8 + }, + { + "id": "hc3-human-0098", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0272, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0062", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0345, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0068", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1207, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0005", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9276, + "verdict": "AI-Generated", + "confidence": 92.8 + }, + { + "id": "hc3-human-0082", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0436, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0006", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0517, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0096", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1846, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0061", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1609, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0099", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.034, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0054", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6853, + "verdict": "Likely AI-Generated", + "confidence": 72.4 + }, + { + "id": "hc3-ai-0042", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9618, + "verdict": "AI-Generated", + "confidence": 93.9 + }, + { + "id": "hc3-human-0046", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1286, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0010", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0178, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0006", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7122, + "verdict": "Likely AI-Generated", + "confidence": 73.5 + }, + { + "id": "hc3-human-0078", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1726, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0050", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0711, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0027", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.89, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-human-0013", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2172, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0086", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0211, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0053", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0622, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0008", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9237, + "verdict": "AI-Generated", + "confidence": 92.7 + }, + { + "id": "hc3-ai-0056", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.882, + "verdict": "AI-Generated", + "confidence": 91.5 + }, + { + "id": "hc3-ai-0069", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.552, + "verdict": "Likely AI-Generated", + "confidence": 67.1 + }, + { + "id": "hc3-human-0031", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1261, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0009", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0653, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0073", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0067, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0028", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.736, + "verdict": "Likely AI-Generated", + "confidence": 74.4 + }, + { + "id": "hc3-ai-0068", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9082, + "verdict": "AI-Generated", + "confidence": 92.2 + }, + { + "id": "hc3-human-0002", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0197, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0000", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8792, + "verdict": "AI-Generated", + "confidence": 91.4 + }, + { + "id": "hc3-ai-0030", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8839, + "verdict": "AI-Generated", + "confidence": 91.5 + }, + { + "id": "hc3-ai-0065", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7486, + "verdict": "Likely AI-Generated", + "confidence": 74.9 + }, + { + "id": "hc3-ai-0078", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8464, + "verdict": "AI-Generated", + "confidence": 90.4 + }, + { + "id": "hc3-ai-0016", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7514, + "verdict": "AI-Generated", + "confidence": 87.5 + }, + { + "id": "hc3-human-0038", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1367, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0036", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0254, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0053", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7085, + "verdict": "Likely AI-Generated", + "confidence": 73.3 + }, + { + "id": "hc3-human-0058", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1458, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0019", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0388, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0026", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3604, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0081", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8899, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-human-0039", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0129, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0016", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0788, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0035", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8278, + "verdict": "AI-Generated", + "confidence": 89.8 + }, + { + "id": "hc3-human-0011", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0473, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0051", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9659, + "verdict": "AI-Generated", + "confidence": 94.0 + }, + { + "id": "hc3-ai-0050", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8223, + "verdict": "AI-Generated", + "confidence": 89.7 + }, + { + "id": "hc3-human-0085", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0179, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0069", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0402, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0023", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8335, + "verdict": "AI-Generated", + "confidence": 90.0 + }, + { + "id": "hc3-human-0008", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4222, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0062", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9581, + "verdict": "AI-Generated", + "confidence": 93.7 + }, + { + "id": "hc3-ai-0022", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9127, + "verdict": "AI-Generated", + "confidence": 92.4 + }, + { + "id": "hc3-human-0014", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0188, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0038", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8334, + "verdict": "AI-Generated", + "confidence": 90.0 + }, + { + "id": "hc3-ai-0073", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8924, + "verdict": "AI-Generated", + "confidence": 91.8 + }, + { + "id": "hc3-human-0024", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2035, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0074", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9132, + "verdict": "AI-Generated", + "confidence": 92.4 + }, + { + "id": "hc3-human-0023", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0532, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0043", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2322, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0072", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.258, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0087", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4785, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0045", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8879, + "verdict": "AI-Generated", + "confidence": 91.6 + }, + { + "id": "hc3-human-0004", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2663, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0047", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8607, + "verdict": "AI-Generated", + "confidence": 90.8 + }, + { + "id": "hc3-ai-0098", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7956, + "verdict": "AI-Generated", + "confidence": 88.9 + }, + { + "id": "hc3-ai-0020", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.911, + "verdict": "AI-Generated", + "confidence": 92.3 + }, + { + "id": "hc3-human-0079", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0371, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0027", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0252, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0089", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7364, + "verdict": "Likely AI-Generated", + "confidence": 74.5 + }, + { + "id": "hc3-ai-0058", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8735, + "verdict": "AI-Generated", + "confidence": 91.2 + }, + { + "id": "hc3-human-0077", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1169, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0049", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9369, + "verdict": "AI-Generated", + "confidence": 93.1 + }, + { + "id": "hc3-human-0017", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0438, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0084", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.071, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0043", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8925, + "verdict": "AI-Generated", + "confidence": 91.8 + }, + { + "id": "hc3-ai-0099", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8074, + "verdict": "AI-Generated", + "confidence": 89.2 + }, + { + "id": "hc3-ai-0088", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8724, + "verdict": "AI-Generated", + "confidence": 91.2 + }, + { + "id": "hc3-ai-0059", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9281, + "verdict": "AI-Generated", + "confidence": 92.8 + }, + { + "id": "hc3-human-0090", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3908, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0025", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3736, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0044", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1035, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0002", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9321, + "verdict": "AI-Generated", + "confidence": 93.0 + }, + { + "id": "hc3-human-0097", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2937, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0021", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7886, + "verdict": "AI-Generated", + "confidence": 88.7 + }, + { + "id": "hc3-human-0094", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0901, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0052", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0805, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0031", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.896, + "verdict": "AI-Generated", + "confidence": 91.9 + }, + { + "id": "hc3-human-0020", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0147, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0079", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8097, + "verdict": "AI-Generated", + "confidence": 89.3 + }, + { + "id": "hc3-human-0081", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4192, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0056", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0955, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0007", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8977, + "verdict": "AI-Generated", + "confidence": 91.9 + }, + { + "id": "hc3-human-0089", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0338, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0003", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6323, + "verdict": "Likely AI-Generated", + "confidence": 70.3 + }, + { + "id": "hc3-human-0063", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0871, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0029", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8852, + "verdict": "AI-Generated", + "confidence": 91.6 + }, + { + "id": "hc3-human-0028", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0851, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0005", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0722, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0092", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9214, + "verdict": "AI-Generated", + "confidence": 92.6 + }, + { + "id": "hc3-ai-0019", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8531, + "verdict": "AI-Generated", + "confidence": 90.6 + }, + { + "id": "hc3-ai-0083", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7041, + "verdict": "Likely AI-Generated", + "confidence": 73.2 + }, + { + "id": "hc3-human-0092", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0066, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0000", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1529, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0032", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7796, + "verdict": "AI-Generated", + "confidence": 88.4 + }, + { + "id": "hc3-ai-0018", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9017, + "verdict": "AI-Generated", + "confidence": 92.1 + }, + { + "id": "hc3-ai-0060", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6263, + "verdict": "Likely AI-Generated", + "confidence": 70.1 + }, + { + "id": "hc3-ai-0085", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9305, + "verdict": "AI-Generated", + "confidence": 92.9 + }, + { + "id": "hc3-ai-0084", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6081, + "verdict": "Likely AI-Generated", + "confidence": 69.3 + }, + { + "id": "hc3-ai-0061", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8848, + "verdict": "AI-Generated", + "confidence": 91.5 + }, + { + "id": "hc3-human-0049", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0261, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0034", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8115, + "verdict": "AI-Generated", + "confidence": 89.3 + }, + { + "id": "hc3-human-0060", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0175, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0025", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8905, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-ai-0086", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8207, + "verdict": "AI-Generated", + "confidence": 89.6 + }, + { + "id": "hc3-ai-0001", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8502, + "verdict": "AI-Generated", + "confidence": 90.5 + }, + { + "id": "hc3-human-0071", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1269, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0077", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8169, + "verdict": "AI-Generated", + "confidence": 89.5 + }, + { + "id": "hc3-ai-0017", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8914, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-human-0095", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.006, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0064", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2155, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0067", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9175, + "verdict": "AI-Generated", + "confidence": 92.5 + }, + { + "id": "hc3-ai-0033", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8465, + "verdict": "AI-Generated", + "confidence": 90.4 + }, + { + "id": "hc3-ai-0048", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6967, + "verdict": "Likely AI-Generated", + "confidence": 72.9 + }, + { + "id": "hc3-human-0051", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0379, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0082", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8884, + "verdict": "AI-Generated", + "confidence": 91.7 + }, + { + "id": "hc3-human-0045", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2082, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0037", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6134, + "verdict": "Likely AI-Generated", + "confidence": 69.5 + }, + { + "id": "hc3-ai-0013", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9309, + "verdict": "AI-Generated", + "confidence": 92.9 + }, + { + "id": "hc3-ai-0012", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8535, + "verdict": "AI-Generated", + "confidence": 90.6 + }, + { + "id": "hc3-human-0034", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0952, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0041", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8316, + "verdict": "AI-Generated", + "confidence": 89.9 + }, + { + "id": "hc3-ai-0014", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9541, + "verdict": "AI-Generated", + "confidence": 93.6 + }, + { + "id": "hc3-ai-0072", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9143, + "verdict": "AI-Generated", + "confidence": 92.4 + }, + { + "id": "hc3-ai-0052", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7548, + "verdict": "AI-Generated", + "confidence": 87.6 + }, + { + "id": "hc3-human-0088", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3418, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0030", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0954, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0080", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0988, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0080", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9374, + "verdict": "AI-Generated", + "confidence": 93.1 + }, + { + "id": "hc3-human-0093", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2183, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0009", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8297, + "verdict": "AI-Generated", + "confidence": 89.9 + }, + { + "id": "hc3-ai-0044", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9583, + "verdict": "AI-Generated", + "confidence": 93.7 + }, + { + "id": "hc3-ai-0040", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9063, + "verdict": "AI-Generated", + "confidence": 92.2 + }, + { + "id": "hc3-ai-0024", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.837, + "verdict": "AI-Generated", + "confidence": 90.1 + }, + { + "id": "hc3-ai-0093", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9339, + "verdict": "AI-Generated", + "confidence": 93.0 + }, + { + "id": "hc3-human-0091", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2899, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-human-0065", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0035", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4459, + "verdict": "Likely Human-Written", + "confidence": 85.0 + }, + { + "id": "hc3-ai-0011", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9024, + "verdict": "AI-Generated", + "confidence": 92.1 + }, + { + "id": "hc3-ai-0015", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9013, + "verdict": "AI-Generated", + "confidence": 92.0 + }, + { + "id": "hc3-human-0067", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.119, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0021", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1491, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0091", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9143, + "verdict": "AI-Generated", + "confidence": 92.4 + }, + { + "id": "hc3-human-0070", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1012, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0003", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0258, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0055", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9579, + "verdict": "AI-Generated", + "confidence": 93.7 + }, + { + "id": "hc3-ai-0096", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9583, + "verdict": "AI-Generated", + "confidence": 93.8 + }, + { + "id": "hc3-human-0032", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0289, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0010", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5814, + "verdict": "Likely AI-Generated", + "confidence": 68.3 + }, + { + "id": "hc3-human-0007", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2365, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0075", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0265, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0054", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0056, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0036", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.898, + "verdict": "AI-Generated", + "confidence": 91.9 + }, + { + "id": "hc3-human-0018", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0289, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0033", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1103, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0095", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8951, + "verdict": "AI-Generated", + "confidence": 91.9 + }, + { + "id": "hc3-ai-0087", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8879, + "verdict": "AI-Generated", + "confidence": 91.6 + }, + { + "id": "hc3-ai-0064", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7718, + "verdict": "AI-Generated", + "confidence": 88.2 + }, + { + "id": "hc3-human-0057", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0732, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0037", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1088, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0070", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8553, + "verdict": "AI-Generated", + "confidence": 90.7 + }, + { + "id": "hc3-human-0059", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0212, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0066", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.9064, + "verdict": "AI-Generated", + "confidence": 92.2 + }, + { + "id": "hc3-human-0047", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1351, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-ai-0097", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8662, + "verdict": "AI-Generated", + "confidence": 91.0 + }, + { + "id": "hc3-ai-0075", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7008, + "verdict": "Likely AI-Generated", + "confidence": 73.0 + }, + { + "id": "hc3-human-0074", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1115, + "verdict": "Human-Written", + "confidence": 95.0 + }, + { + "id": "hc3-human-0066", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1929, + "verdict": "Human-Written", + "confidence": 95.0 + } + ] +} \ No newline at end of file diff --git a/docs/benchmarks/hc3_ensemble_report.json b/docs/benchmarks/hc3_ensemble_report.json new file mode 100644 index 0000000..350ca38 --- /dev/null +++ b/docs/benchmarks/hc3_ensemble_report.json @@ -0,0 +1,1717 @@ +{ + "analyzer": "ensemble", + "n_samples": 200, + "metrics_at_0.5": { + "threshold": 0.5, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 100, + "false_positives": 10, + "true_negatives": 90, + "false_negatives": 0, + "accuracy": 0.95, + "precision": 0.9090909090909091, + "recall": 1.0, + "f1": 0.9523809523809523, + "specificity": 0.9, + "false_positive_rate": 0.1, + "false_negative_rate": 0.0, + "roc_auc": 0.998, + "expected_calibration_error": 0.20068447995311733 + }, + "best_f1_threshold": 0.7246, + "metrics_at_best_f1": { + "threshold": 0.7246038952078104, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 99, + "false_positives": 1, + "true_negatives": 99, + "false_negatives": 1, + "accuracy": 0.99, + "precision": 0.99, + "recall": 0.99, + "f1": 0.99, + "specificity": 0.99, + "false_positive_rate": 0.01, + "false_negative_rate": 0.01, + "roc_auc": 0.998, + "expected_calibration_error": 0.20068447995311733 + }, + "calibration_bins": [ + { + "bin_lower": 0.0, + "bin_upper": 0.1, + "count": 38, + "mean_predicted": 0.06919958049337709, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.1, + "bin_upper": 0.2, + "count": 31, + "mean_predicted": 0.14249968555756998, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.2, + "bin_upper": 0.30000000000000004, + "count": 12, + "mean_predicted": 0.24634711872289447, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.30000000000000004, + "bin_upper": 0.4, + "count": 6, + "mean_predicted": 0.37647048473835265, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.4, + "bin_upper": 0.5, + "count": 3, + "mean_predicted": 0.4173270964705574, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.5, + "bin_upper": 0.6000000000000001, + "count": 7, + "mean_predicted": 0.552372130005687, + "observed_fraction": 0.14285714285714285 + }, + { + "bin_lower": 0.6000000000000001, + "bin_upper": 0.7000000000000001, + "count": 3, + "mean_predicted": 0.6679538452731187, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.7000000000000001, + "bin_upper": 0.8, + "count": 92, + "mean_predicted": 0.7688938657024901, + "observed_fraction": 0.9891304347826086 + }, + { + "bin_lower": 0.8, + "bin_upper": 0.9, + "count": 8, + "mean_predicted": 0.8136723430195161, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.9, + "bin_upper": 1.0, + "count": 0, + "mean_predicted": NaN, + "observed_fraction": NaN + } + ], + "predictions": [ + { + "id": "hc3-human-0055", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1475, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0012", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1482, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0076", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0932, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0076", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8006, + "verdict": "AI-Generated", + "confidence": 87.7 + }, + { + "id": "hc3-ai-0063", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.509, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0071", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7649, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0015", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4113, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-human-0029", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0873, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0040", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.057, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0039", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7536, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-human-0001", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1381, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0041", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1249, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0094", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7531, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-ai-0057", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7973, + "verdict": "AI-Generated", + "confidence": 87.6 + }, + { + "id": "hc3-ai-0026", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7714, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-human-0048", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.097, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0022", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0539, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0083", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1063, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0004", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.775, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-human-0042", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1826, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0090", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7555, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-ai-0046", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7805, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-human-0098", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0623, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0062", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0392, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0068", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1879, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0005", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7666, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0082", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2263, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0006", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1067, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0096", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.408, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-human-0061", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1438, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0099", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.064, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0054", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7891, + "verdict": "AI-Generated", + "confidence": 87.4 + }, + { + "id": "hc3-ai-0042", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7609, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0046", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1838, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0010", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0613, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0006", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7611, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0078", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2541, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0050", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.106, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0027", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7671, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0013", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3578, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-human-0086", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0997, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0053", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1178, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0008", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7924, + "verdict": "AI-Generated", + "confidence": 87.5 + }, + { + "id": "hc3-ai-0056", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7829, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-ai-0069", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7894, + "verdict": "AI-Generated", + "confidence": 87.4 + }, + { + "id": "hc3-human-0031", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5108, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0009", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1458, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0073", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0756, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0028", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7632, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-ai-0068", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7628, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0002", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0814, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0000", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7612, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-ai-0030", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7792, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-ai-0065", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7588, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0078", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.759, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0016", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7582, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0038", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1508, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0036", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0606, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0053", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7246, + "verdict": "AI-Generated", + "confidence": 85.7 + }, + { + "id": "hc3-human-0058", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1514, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0019", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.097, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0026", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2836, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0081", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7782, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-human-0039", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0896, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0016", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0577, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0035", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7639, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0011", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0727, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0051", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7662, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0050", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7554, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-human-0085", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0595, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0069", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0487, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0023", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7833, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-human-0008", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6767, + "verdict": "Likely AI-Generated", + "confidence": 71.0 + }, + { + "id": "hc3-ai-0062", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7646, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-ai-0022", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7752, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-human-0014", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0678, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0038", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8039, + "verdict": "AI-Generated", + "confidence": 87.8 + }, + { + "id": "hc3-ai-0073", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7668, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0024", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2212, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0074", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7789, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-human-0023", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.044, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0043", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2612, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0072", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0892, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0087", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7569, + "verdict": "AI-Generated", + "confidence": 87.7 + }, + { + "id": "hc3-ai-0045", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8034, + "verdict": "AI-Generated", + "confidence": 87.7 + }, + { + "id": "hc3-human-0004", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3327, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0047", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7505, + "verdict": "AI-Generated", + "confidence": 86.4 + }, + { + "id": "hc3-ai-0098", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7958, + "verdict": "AI-Generated", + "confidence": 87.5 + }, + { + "id": "hc3-ai-0020", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7659, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0079", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0923, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0027", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1573, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0089", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7682, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0058", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8141, + "verdict": "AI-Generated", + "confidence": 88.0 + }, + { + "id": "hc3-human-0077", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1566, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0049", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7661, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0017", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1168, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0084", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1272, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0043", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7651, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0099", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7662, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0088", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7675, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0059", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7608, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0090", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5748, + "verdict": "Likely AI-Generated", + "confidence": 67.5 + }, + { + "id": "hc3-human-0025", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5549, + "verdict": "Likely AI-Generated", + "confidence": 66.9 + }, + { + "id": "hc3-human-0044", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.075, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0002", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7623, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0097", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2577, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0021", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7722, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-human-0094", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0762, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0052", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0389, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0031", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7768, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-human-0020", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0698, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0079", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.782, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-human-0081", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5467, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0056", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0817, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0007", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7592, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0089", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0653, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0003", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7466, + "verdict": "AI-Generated", + "confidence": 86.3 + }, + { + "id": "hc3-human-0063", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1124, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0029", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7785, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-human-0028", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1446, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0005", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0936, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0092", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8287, + "verdict": "AI-Generated", + "confidence": 88.4 + }, + { + "id": "hc3-ai-0019", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7815, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-ai-0083", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7727, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-human-0092", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.087, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0000", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2006, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0032", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8348, + "verdict": "AI-Generated", + "confidence": 88.5 + }, + { + "id": "hc3-ai-0018", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.773, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-ai-0060", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7687, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-ai-0085", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7579, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0084", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7586, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0061", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7703, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-human-0049", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0552, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0034", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8171, + "verdict": "AI-Generated", + "confidence": 88.1 + }, + { + "id": "hc3-human-0060", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0553, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0025", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7872, + "verdict": "AI-Generated", + "confidence": 87.3 + }, + { + "id": "hc3-ai-0086", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.768, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0001", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7814, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-human-0071", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3904, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0077", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7902, + "verdict": "AI-Generated", + "confidence": 87.4 + }, + { + "id": "hc3-ai-0017", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7597, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0095", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0456, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0064", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1616, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0067", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7652, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0033", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7716, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-ai-0048", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7577, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0051", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1793, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0082", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7608, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0045", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.4327, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0037", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7775, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-ai-0013", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7744, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-ai-0012", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7651, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0034", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1005, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0041", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7809, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-ai-0014", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7673, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0072", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7591, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0052", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7783, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-human-0088", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6423, + "verdict": "Likely AI-Generated", + "confidence": 69.8 + }, + { + "id": "hc3-human-0030", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1285, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0080", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3969, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0080", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7556, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-human-0093", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6849, + "verdict": "Likely AI-Generated", + "confidence": 71.3 + }, + { + "id": "hc3-ai-0009", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7811, + "verdict": "AI-Generated", + "confidence": 87.2 + }, + { + "id": "hc3-ai-0044", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7799, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-ai-0040", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7668, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0024", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7775, + "verdict": "AI-Generated", + "confidence": 87.1 + }, + { + "id": "hc3-ai-0093", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7669, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0091", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5972, + "verdict": "Likely AI-Generated", + "confidence": 68.3 + }, + { + "id": "hc3-human-0065", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2746, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0035", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5733, + "verdict": "Likely AI-Generated", + "confidence": 67.5 + }, + { + "id": "hc3-ai-0011", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8067, + "verdict": "AI-Generated", + "confidence": 87.8 + }, + { + "id": "hc3-ai-0015", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7944, + "verdict": "AI-Generated", + "confidence": 87.5 + }, + { + "id": "hc3-human-0067", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2622, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0021", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3918, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0091", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7719, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-human-0070", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.107, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0003", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0652, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0055", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7601, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-ai-0096", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.787, + "verdict": "AI-Generated", + "confidence": 87.3 + }, + { + "id": "hc3-human-0032", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0457, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0010", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7422, + "verdict": "AI-Generated", + "confidence": 86.2 + }, + { + "id": "hc3-human-0007", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2325, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0075", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0793, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0054", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.0448, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0036", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7747, + "verdict": "AI-Generated", + "confidence": 87.0 + }, + { + "id": "hc3-human-0018", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1475, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0033", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1251, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0095", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7669, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0087", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7567, + "verdict": "AI-Generated", + "confidence": 86.5 + }, + { + "id": "hc3-ai-0064", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7617, + "verdict": "AI-Generated", + "confidence": 86.7 + }, + { + "id": "hc3-human-0057", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1786, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0037", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3893, + "verdict": "Likely Human-Written", + "confidence": 85 + }, + { + "id": "hc3-ai-0070", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7608, + "verdict": "AI-Generated", + "confidence": 86.6 + }, + { + "id": "hc3-human-0059", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1371, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0066", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7721, + "verdict": "AI-Generated", + "confidence": 86.9 + }, + { + "id": "hc3-human-0047", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2652, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-ai-0097", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7657, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-ai-0075", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7685, + "verdict": "AI-Generated", + "confidence": 86.8 + }, + { + "id": "hc3-human-0074", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.217, + "verdict": "Human-Written", + "confidence": 95 + }, + { + "id": "hc3-human-0066", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1956, + "verdict": "Human-Written", + "confidence": 95 + } + ] +} \ No newline at end of file diff --git a/docs/benchmarks/hc3_gpt2_report.json b/docs/benchmarks/hc3_gpt2_report.json new file mode 100644 index 0000000..a29a23b --- /dev/null +++ b/docs/benchmarks/hc3_gpt2_report.json @@ -0,0 +1,1717 @@ +{ + "analyzer": "gpt2", + "n_samples": 200, + "metrics_at_0.5": { + "threshold": 0.5, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 100, + "false_positives": 50, + "true_negatives": 50, + "false_negatives": 0, + "accuracy": 0.75, + "precision": 0.6666666666666666, + "recall": 1.0, + "f1": 0.8, + "specificity": 0.5, + "false_positive_rate": 0.5, + "false_negative_rate": 0.0, + "roc_auc": 0.7561, + "expected_calibration_error": 0.170185 + }, + "best_f1_threshold": 0.5, + "metrics_at_best_f1": { + "threshold": 0.5, + "n_samples": 200, + "n_positive": 100, + "n_negative": 100, + "true_positives": 100, + "false_positives": 50, + "true_negatives": 50, + "false_negatives": 0, + "accuracy": 0.75, + "precision": 0.6666666666666666, + "recall": 1.0, + "f1": 0.8, + "specificity": 0.5, + "false_positive_rate": 0.5, + "false_negative_rate": 0.0, + "roc_auc": 0.7561, + "expected_calibration_error": 0.170185 + }, + "calibration_bins": [ + { + "bin_lower": 0.0, + "bin_upper": 0.1, + "count": 1, + "mean_predicted": 0.09899999999999998, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.1, + "bin_upper": 0.2, + "count": 33, + "mean_predicted": 0.16943939393939397, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.2, + "bin_upper": 0.30000000000000004, + "count": 14, + "mean_predicted": 0.24432142857142858, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.30000000000000004, + "bin_upper": 0.4, + "count": 2, + "mean_predicted": 0.3015, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.4, + "bin_upper": 0.5, + "count": 0, + "mean_predicted": NaN, + "observed_fraction": NaN + }, + { + "bin_lower": 0.5, + "bin_upper": 0.6000000000000001, + "count": 131, + "mean_predicted": 0.5, + "observed_fraction": 0.6641221374045801 + }, + { + "bin_lower": 0.6000000000000001, + "bin_upper": 0.7000000000000001, + "count": 1, + "mean_predicted": 0.6995, + "observed_fraction": 0.0 + }, + { + "bin_lower": 0.7000000000000001, + "bin_upper": 0.8, + "count": 15, + "mean_predicted": 0.7682333333333333, + "observed_fraction": 0.6666666666666666 + }, + { + "bin_lower": 0.8, + "bin_upper": 0.9, + "count": 3, + "mean_predicted": 0.8000000000000002, + "observed_fraction": 1.0 + }, + { + "bin_lower": 0.9, + "bin_upper": 1.0, + "count": 0, + "mean_predicted": NaN, + "observed_fraction": NaN + } + ], + "predictions": [ + { + "id": "hc3-human-0055", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 42.6 + }, + { + "id": "hc3-human-0012", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0076", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.737, + "verdict": "Likely AI-Generated", + "confidence": 47.4 + }, + { + "id": "hc3-ai-0076", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0063", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.776, + "verdict": "Likely AI-Generated", + "confidence": 55.2 + }, + { + "id": "hc3-ai-0071", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0015", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 32.8 + }, + { + "id": "hc3-human-0029", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2, + "verdict": "Likely Human-Written", + "confidence": 60.0 + }, + { + "id": "hc3-human-0040", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1965, + "verdict": "Likely Human-Written", + "confidence": 60.7 + }, + { + "id": "hc3-ai-0039", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0001", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.285, + "verdict": "Likely Human-Written", + "confidence": 43.0 + }, + { + "id": "hc3-human-0041", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 42.0 + }, + { + "id": "hc3-ai-0094", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8, + "verdict": "Likely AI-Generated", + "confidence": 60.0 + }, + { + "id": "hc3-ai-0057", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.785, + "verdict": "Likely AI-Generated", + "confidence": 57.0 + }, + { + "id": "hc3-ai-0026", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0048", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7165, + "verdict": "Likely AI-Generated", + "confidence": 43.3 + }, + { + "id": "hc3-human-0022", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.233, + "verdict": "Likely Human-Written", + "confidence": 53.4 + }, + { + "id": "hc3-human-0083", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.227, + "verdict": "Likely Human-Written", + "confidence": 54.6 + }, + { + "id": "hc3-ai-0004", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.785, + "verdict": "Likely AI-Generated", + "confidence": 57.0 + }, + { + "id": "hc3-human-0042", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.171, + "verdict": "Likely Human-Written", + "confidence": 65.8 + }, + { + "id": "hc3-ai-0090", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0046", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.785, + "verdict": "Likely AI-Generated", + "confidence": 57.0 + }, + { + "id": "hc3-human-0098", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.215, + "verdict": "Likely Human-Written", + "confidence": 57.0 + }, + { + "id": "hc3-human-0062", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2695, + "verdict": "Likely Human-Written", + "confidence": 46.1 + }, + { + "id": "hc3-human-0068", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0005", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0082", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0006", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 40.3 + }, + { + "id": "hc3-human-0096", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.777, + "verdict": "Likely AI-Generated", + "confidence": 55.4 + }, + { + "id": "hc3-human-0061", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0099", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3005, + "verdict": "Likely Human-Written", + "confidence": 39.9 + }, + { + "id": "hc3-ai-0054", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0042", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0046", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0010", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1965, + "verdict": "Likely Human-Written", + "confidence": 60.7 + }, + { + "id": "hc3-ai-0006", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0078", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.194, + "verdict": "Likely Human-Written", + "confidence": 61.2 + }, + { + "id": "hc3-human-0050", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2245, + "verdict": "Likely Human-Written", + "confidence": 55.1 + }, + { + "id": "hc3-ai-0027", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0013", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0086", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.114, + "verdict": "Human-Written", + "confidence": 77.2 + }, + { + "id": "hc3-human-0053", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.125, + "verdict": "Human-Written", + "confidence": 75.0 + }, + { + "id": "hc3-ai-0008", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0056", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0069", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0031", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.195, + "verdict": "Likely Human-Written", + "confidence": 61.0 + }, + { + "id": "hc3-human-0009", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0073", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.197, + "verdict": "Likely Human-Written", + "confidence": 60.6 + }, + { + "id": "hc3-ai-0028", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0068", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0002", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0000", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0030", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0065", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0078", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0016", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8, + "verdict": "Likely AI-Generated", + "confidence": 60.0 + }, + { + "id": "hc3-human-0038", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0036", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 36.1 + }, + { + "id": "hc3-ai-0053", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0058", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.171, + "verdict": "Likely Human-Written", + "confidence": 65.8 + }, + { + "id": "hc3-human-0019", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0026", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 41.7 + }, + { + "id": "hc3-ai-0081", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0039", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.273, + "verdict": "Likely Human-Written", + "confidence": 45.4 + }, + { + "id": "hc3-human-0016", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.11, + "verdict": "Human-Written", + "confidence": 78.0 + }, + { + "id": "hc3-ai-0035", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.776, + "verdict": "Likely AI-Generated", + "confidence": 55.2 + }, + { + "id": "hc3-human-0011", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0051", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0050", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0085", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.138, + "verdict": "Human-Written", + "confidence": 72.4 + }, + { + "id": "hc3-human-0069", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2075, + "verdict": "Likely Human-Written", + "confidence": 58.5 + }, + { + "id": "hc3-ai-0023", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0008", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0062", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0022", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0014", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0038", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0073", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0024", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0074", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0023", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2955, + "verdict": "Likely Human-Written", + "confidence": 40.9 + }, + { + "id": "hc3-human-0043", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0072", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0087", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7465, + "verdict": "Likely AI-Generated", + "confidence": 49.3 + }, + { + "id": "hc3-ai-0045", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0004", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0047", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0098", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0020", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0079", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.153, + "verdict": "Likely Human-Written", + "confidence": 69.4 + }, + { + "id": "hc3-human-0027", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-ai-0089", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0058", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0077", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 32.4 + }, + { + "id": "hc3-ai-0049", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0017", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1565, + "verdict": "Likely Human-Written", + "confidence": 68.7 + }, + { + "id": "hc3-human-0084", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.125, + "verdict": "Likely Human-Written", + "confidence": 75.0 + }, + { + "id": "hc3-ai-0043", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0099", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0088", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0059", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0090", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 35.6 + }, + { + "id": "hc3-human-0025", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0044", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-ai-0002", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0097", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 43.0 + }, + { + "id": "hc3-ai-0021", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0094", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.099, + "verdict": "Human-Written", + "confidence": 80.2 + }, + { + "id": "hc3-human-0052", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2775, + "verdict": "Likely Human-Written", + "confidence": 44.5 + }, + { + "id": "hc3-ai-0031", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.785, + "verdict": "Likely AI-Generated", + "confidence": 57.0 + }, + { + "id": "hc3-human-0020", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1785, + "verdict": "Likely Human-Written", + "confidence": 64.3 + }, + { + "id": "hc3-ai-0079", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0081", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.731, + "verdict": "Likely AI-Generated", + "confidence": 46.2 + }, + { + "id": "hc3-human-0056", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0007", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0089", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1755, + "verdict": "Likely Human-Written", + "confidence": 64.9 + }, + { + "id": "hc3-ai-0003", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.776, + "verdict": "Likely AI-Generated", + "confidence": 55.2 + }, + { + "id": "hc3-human-0063", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0029", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0028", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0005", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0092", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0019", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0083", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0092", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.145, + "verdict": "Likely Human-Written", + "confidence": 71.0 + }, + { + "id": "hc3-human-0000", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.171, + "verdict": "Likely Human-Written", + "confidence": 65.8 + }, + { + "id": "hc3-ai-0032", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0018", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0060", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0085", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0084", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0061", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0049", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 47.1 + }, + { + "id": "hc3-ai-0034", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0060", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2445, + "verdict": "Likely Human-Written", + "confidence": 51.1 + }, + { + "id": "hc3-ai-0025", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0086", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0001", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0071", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0077", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.8, + "verdict": "Likely AI-Generated", + "confidence": 60.0 + }, + { + "id": "hc3-ai-0017", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0095", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.3025, + "verdict": "Likely Human-Written", + "confidence": 39.5 + }, + { + "id": "hc3-human-0064", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0067", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0033", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0048", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0051", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0082", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0045", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1865, + "verdict": "Likely Human-Written", + "confidence": 62.7 + }, + { + "id": "hc3-ai-0037", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.781, + "verdict": "Likely AI-Generated", + "confidence": 56.2 + }, + { + "id": "hc3-ai-0013", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0012", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0034", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 46.5 + }, + { + "id": "hc3-ai-0041", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0014", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0072", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0052", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0088", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0030", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0080", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0080", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0093", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 42.1 + }, + { + "id": "hc3-ai-0009", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0044", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0040", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.7815, + "verdict": "Likely AI-Generated", + "confidence": 56.3 + }, + { + "id": "hc3-ai-0024", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0093", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0091", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0065", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0035", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0011", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0015", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0067", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0021", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.171, + "verdict": "Likely Human-Written", + "confidence": 65.8 + }, + { + "id": "hc3-ai-0091", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0070", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0003", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.13, + "verdict": "Likely Human-Written", + "confidence": 74.0 + }, + { + "id": "hc3-ai-0055", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0096", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0032", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.1815, + "verdict": "Likely Human-Written", + "confidence": 63.7 + }, + { + "id": "hc3-ai-0010", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0007", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0075", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.232, + "verdict": "Likely Human-Written", + "confidence": 53.6 + }, + { + "id": "hc3-human-0054", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.2365, + "verdict": "Likely Human-Written", + "confidence": 52.7 + }, + { + "id": "hc3-ai-0036", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0018", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.11, + "verdict": "Human-Written", + "confidence": 78.0 + }, + { + "id": "hc3-human-0033", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 34.4 + }, + { + "id": "hc3-ai-0095", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0087", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0064", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0057", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.6995, + "verdict": "Likely AI-Generated", + "confidence": 39.9 + }, + { + "id": "hc3-human-0037", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-ai-0070", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.785, + "verdict": "Likely AI-Generated", + "confidence": 57.0 + }, + { + "id": "hc3-human-0059", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0066", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0047", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0097", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-ai-0075", + "label": 1, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + }, + { + "id": "hc3-human-0074", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.19, + "verdict": "Likely Human-Written", + "confidence": 62.0 + }, + { + "id": "hc3-human-0066", + "label": 0, + "source": "hc3-reddit_eli5", + "ai_probability": 0.5, + "verdict": "Uncertain", + "confidence": 50.0 + } + ] +} \ No newline at end of file diff --git a/docs/benchmarks/roc_hc3_binoculars.png b/docs/benchmarks/roc_hc3_binoculars.png new file mode 100644 index 0000000..83cafb9 Binary files /dev/null and b/docs/benchmarks/roc_hc3_binoculars.png differ diff --git a/docs/benchmarks/roc_hc3_ensemble.png b/docs/benchmarks/roc_hc3_ensemble.png new file mode 100644 index 0000000..17a1045 Binary files /dev/null and b/docs/benchmarks/roc_hc3_ensemble.png differ diff --git a/docs/benchmarks/roc_hc3_gpt2.png b/docs/benchmarks/roc_hc3_gpt2.png new file mode 100644 index 0000000..7454fb8 Binary files /dev/null and b/docs/benchmarks/roc_hc3_gpt2.png differ diff --git a/scripts/calibrate_binoculars.py b/scripts/calibrate_binoculars.py new file mode 100644 index 0000000..c773e7e --- /dev/null +++ b/scripts/calibrate_binoculars.py @@ -0,0 +1,127 @@ +""" +Calibrate the Binoculars decision boundary on a labelled corpus +=============================================================== + +The Binoculars *ratio* (observer log-perplexity / cross-perplexity) ranks +human vs machine text extremely well, but the value of the decision boundary is +corpus-dependent. ``BinocularsConfig.score_midpoint`` was originally fit on the +tiny bundled benchmark and does not transfer to real-world text. + +This script fits the boundary **honestly**: + +1. Compute the raw ratio for every sample (cached to JSON so re-runs are free). +2. Split the corpus 50/50, stratified by label, with a fixed seed. +3. Sweep candidate midpoints on the **calibration half** and pick the one with + the best balanced accuracy. +4. Report performance on the **held-out half** — the number you can trust. + +Usage +----- + python scripts/calibrate_binoculars.py --dataset data/external/hc3_sample.jsonl +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Dict, List, Tuple + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from src.evaluation.dataset import load_dataset # noqa: E402 + + +def compute_ratios(dataset: str, cache_path: Path) -> List[Dict]: + """Compute (or load cached) Binoculars ratios for every sample.""" + if cache_path.exists(): + print(f"Using cached ratios: {cache_path}") + return json.loads(cache_path.read_text(encoding="utf-8")) + + from src.analyzers.binoculars_analyzer import BinocularsAnalyzer + + analyzer = BinocularsAnalyzer() + rows = [] + samples = load_dataset(dataset) + for i, sample in enumerate(samples, 1): + ratio, _ = analyzer._compute_binoculars(sample.text) + rows.append({"id": sample.id, "label": sample.label, "ratio": ratio}) + if i % 25 == 0: + print(f" {i}/{len(samples)} scored") + + cache_path.parent.mkdir(parents=True, exist_ok=True) + cache_path.write_text(json.dumps(rows, indent=2), encoding="utf-8") + print(f"Cached ratios -> {cache_path}") + return rows + + +def stratified_split(rows: List[Dict], seed: int) -> Tuple[List[Dict], List[Dict]]: + """Split 50/50 preserving class balance.""" + import random + + rng = random.Random(seed) + calib, held = [], [] + for label in (0, 1): + group = [r for r in rows if r["label"] == label] + rng.shuffle(group) + mid = len(group) // 2 + calib.extend(group[:mid]) + held.extend(group[mid:]) + return calib, held + + +def balanced_accuracy(rows: List[Dict], midpoint: float) -> float: + """Lower ratio => AI. Balanced accuracy at a candidate midpoint.""" + tp = sum(1 for r in rows if r["label"] == 1 and r["ratio"] < midpoint) + fn = sum(1 for r in rows if r["label"] == 1 and r["ratio"] >= midpoint) + tn = sum(1 for r in rows if r["label"] == 0 and r["ratio"] >= midpoint) + fp = sum(1 for r in rows if r["label"] == 0 and r["ratio"] < midpoint) + tpr = tp / (tp + fn) if (tp + fn) else 0.0 + tnr = tn / (tn + fp) if (tn + fp) else 0.0 + return (tpr + tnr) / 2 + + +def report(rows: List[Dict], midpoint: float, name: str) -> None: + tp = sum(1 for r in rows if r["label"] == 1 and r["ratio"] < midpoint) + fn = sum(1 for r in rows if r["label"] == 1 and r["ratio"] >= midpoint) + tn = sum(1 for r in rows if r["label"] == 0 and r["ratio"] >= midpoint) + fp = sum(1 for r in rows if r["label"] == 0 and r["ratio"] < midpoint) + n = len(rows) + print( + f"\n{name} (n={n}, midpoint={midpoint:.4f}):\n" + f" accuracy = {(tp + tn) / n:.3f}\n" + f" FPR (human flagged AI) = {fp / (fp + tn) if (fp + tn) else 0:.3f}\n" + f" FNR (AI missed) = {fn / (fn + tp) if (fn + tp) else 0:.3f}" + ) + + +def main(argv: List[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Fit the Binoculars decision boundary.") + parser.add_argument("--dataset", default="data/external/hc3_sample.jsonl") + parser.add_argument("--cache", default="data/external/hc3_binoculars_ratios.json") + parser.add_argument("--seed", type=int, default=13) + args = parser.parse_args(argv) + + rows = compute_ratios(args.dataset, Path(args.cache)) + human = sorted(r["ratio"] for r in rows if r["label"] == 0) + ai = sorted(r["ratio"] for r in rows if r["label"] == 1) + print(f"\nratio human: min={human[0]:.4f} med={human[len(human)//2]:.4f} max={human[-1]:.4f}") + print(f"ratio ai : min={ai[0]:.4f} med={ai[len(ai)//2]:.4f} max={ai[-1]:.4f}") + + calib, held = stratified_split(rows, args.seed) + + # Sweep candidate midpoints between the observed extremes. + lo, hi = min(r["ratio"] for r in rows), max(r["ratio"] for r in rows) + candidates = [lo + (hi - lo) * i / 2000 for i in range(2001)] + best_mid = max(candidates, key=lambda m: balanced_accuracy(calib, m)) + + print(f"\nFitted on calibration half (n={len(calib)}): midpoint = {best_mid:.4f}") + report(calib, best_mid, "CALIBRATION half") + report(held, best_mid, "HELD-OUT half <-- trust this") + print(f"\nSet BinocularsConfig.score_midpoint = {best_mid:.3f}") + return 0 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/scripts/prepare_hc3.py b/scripts/prepare_hc3.py new file mode 100644 index 0000000..c1e31a3 --- /dev/null +++ b/scripts/prepare_hc3.py @@ -0,0 +1,141 @@ +""" +Prepare an HC3 evaluation set +============================= + +Downloads a slice of the public **HC3** corpus (Human ChatGPT Comparison Corpus, +``Hello-SimpleAI/HC3``) and converts it into the JSONL schema this project's +benchmark runner expects:: + + {"id": ..., "label": "human"|"ai", "source": ..., "text": ...} + +Why this exists +--------------- +The bundled ``data/benchmark/samples.jsonl`` is 24 clean, in-distribution +samples — useful for regression/calibration checks, useless as an accuracy +claim. HC3 is a large, real, out-of-distribution corpus (real human answers from +Reddit/finance/medicine/open-QA vs. ChatGPT answers), so it gives the detector an +honest test. + +The downloaded corpus is **not committed** (third-party data, own licence). Only +this script and the resulting metrics live in the repo, so anyone can reproduce. + +Usage +----- + python scripts/prepare_hc3.py # 100 per class + python scripts/prepare_hc3.py --per-class 250 --split reddit_eli5 + +Then benchmark against it: + + python -m src.evaluation.benchmark --analyzer ensemble \ + --dataset data/external/hc3_sample.jsonl +""" + +from __future__ import annotations + +import argparse +import json +import random +import urllib.request +from pathlib import Path +from typing import Iterator, List + +HC3_BASE = "https://huggingface.co/datasets/Hello-SimpleAI/HC3/resolve/main" +DEFAULT_OUT = Path("data/external/hc3_sample.jsonl") + +# Splits available upstream. "all" mixes every domain. +SPLITS = ("all", "reddit_eli5", "open_qa", "finance", "medicine") + + +def _stream_lines(url: str) -> Iterator[str]: + """Stream a remote JSONL file line by line (so we can stop early).""" + request = urllib.request.Request(url, headers={"User-Agent": "ai-text-detector/2.0"}) + with urllib.request.urlopen(request, timeout=60) as response: + for raw in response: + yield raw.decode("utf-8", errors="replace") + + +def _clean(text: str) -> str: + """Collapse whitespace; HC3 answers contain newlines and stray spacing.""" + return " ".join(text.split()).strip() + + +def build_samples(split: str, per_class: int, min_chars: int, seed: int) -> List[dict]: + """Collect a balanced human/AI sample from the HC3 split.""" + url = f"{HC3_BASE}/{split}.jsonl" + human: List[dict] = [] + ai: List[dict] = [] + + for line in _stream_lines(url): + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + + source = record.get("source", split) + + if len(human) < per_class: + for answer in record.get("human_answers") or []: + text = _clean(answer) + if len(text) >= min_chars: + human.append({"label": "human", "source": f"hc3-{source}", "text": text}) + break + + if len(ai) < per_class: + for answer in record.get("chatgpt_answers") or []: + text = _clean(answer) + if len(text) >= min_chars: + ai.append({"label": "ai", "source": f"hc3-{source}", "text": text}) + break + + if len(human) >= per_class and len(ai) >= per_class: + break + + if len(human) < per_class or len(ai) < per_class: + raise RuntimeError( + f"Only collected {len(human)} human / {len(ai)} ai samples " + f"(wanted {per_class} each). Try a different --split or lower --min-chars." + ) + + samples = [] + for i, record in enumerate(human): + samples.append({"id": f"hc3-human-{i:04d}", **record}) + for i, record in enumerate(ai): + samples.append({"id": f"hc3-ai-{i:04d}", **record}) + + random.Random(seed).shuffle(samples) + return samples + + +def main(argv: List[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Build an HC3 evaluation JSONL.") + parser.add_argument("--split", default="all", choices=SPLITS) + parser.add_argument("--per-class", type=int, default=100) + parser.add_argument( + "--min-chars", type=int, default=200, help="matches the tool's recommended minimum" + ) + parser.add_argument("--seed", type=int, default=13) + parser.add_argument("--output", default=str(DEFAULT_OUT)) + args = parser.parse_args(argv) + + print(f"Streaming HC3 split={args.split} ...") + samples = build_samples(args.split, args.per_class, args.min_chars, args.seed) + + out_path = Path(args.output) + out_path.parent.mkdir(parents=True, exist_ok=True) + with out_path.open("w", encoding="utf-8") as fh: + for sample in samples: + fh.write(json.dumps(sample, ensure_ascii=False) + "\n") + + n_ai = sum(1 for s in samples if s["label"] == "ai") + print(f"Wrote {len(samples)} samples ({len(samples) - n_ai} human / {n_ai} ai) to {out_path}") + print( + f"Benchmark it:\n python -m src.evaluation.benchmark --analyzer ensemble --dataset {out_path}" + ) + return 0 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/src/config/settings.py b/src/config/settings.py index dba7013..053cb30 100644 --- a/src/config/settings.py +++ b/src/config/settings.py @@ -139,9 +139,16 @@ class BinocularsConfig: performer_model: str = "distilgpt2" max_token_length: int = 512 revision: Optional[str] = None - # Lower ratio => more AI. Midpoint calibrated from the benchmark separation - # (human scores ~0.88-1.05, AI ~0.72-0.84; boundary between the clusters). - score_midpoint: float = 0.863 + # Lower ratio => more AI. Midpoint fitted on the HC3 corpus (real human text + # vs real ChatGPT output): human ratios ~0.77-1.10, AI ~0.60-0.76. Fitted on a + # calibration half and validated on a held-out half (accuracy 1.000, FPR + # 0.000) via scripts/calibrate_binoculars.py. + # + # NOTE: the previous value (0.863) was fitted on the tiny bundled benchmark, + # whose "AI" samples are hand-written imitations of LLM style rather than real + # model output. It sat inside the real human cluster and flagged 46% of real + # human text as AI. Re-fit on real LLM output before trusting any new value. + score_midpoint: float = 0.7625 score_slope: float = 20.0 diff --git a/src/evaluation/benchmark.py b/src/evaluation/benchmark.py index 67f887f..ef74283 100644 --- a/src/evaluation/benchmark.py +++ b/src/evaluation/benchmark.py @@ -37,6 +37,10 @@ _AI_VERDICTS = (Verdict.AI_GENERATED, Verdict.LIKELY_AI) _HUMAN_VERDICTS = (Verdict.HUMAN_WRITTEN, Verdict.LIKELY_HUMAN) +# Analyzers that publish a calibrated AI-probability as their primary score. +# Checked in order; the first match wins. +_PRIMARY_SCORE_NAMES = ("Ensemble AI Score", "Binoculars AI Score") + class _Analyzer(Protocol): def analyze(self, text: str) -> AnalysisResult: ... @@ -45,11 +49,15 @@ def analyze(self, text: str) -> AnalysisResult: ... def result_to_ai_probability(result: AnalysisResult) -> float: """Map an :class:`AnalysisResult` to a single AI-probability in ``[0, 1]``. - Prefers an explicit ensemble AI score; otherwise derives a monotone score - from the verdict direction and confidence. + Prefers an analyzer's own **calibrated** primary score when it exposes one + (the ensemble and Binoculars do). Only analyzers without a calibrated + probability fall back to a coarse monotone score derived from the verdict + direction and confidence — that fallback is a step function, so it distorts + calibration metrics (ECE) and threshold sweeps. """ - for score in result.scores: - if score.name == "Ensemble AI Score": + for name in _PRIMARY_SCORE_NAMES: + score = result.get_score(name) + if score is not None: return max(0.0, min(1.0, float(score.value))) conf = max(0.0, min(1.0, result.confidence / 100.0)) diff --git a/tests/test_evaluation.py b/tests/test_evaluation.py index 75abc9d..5bac03a 100644 --- a/tests/test_evaluation.py +++ b/tests/test_evaluation.py @@ -98,6 +98,19 @@ def test_prefers_ensemble_score(self): result.add_score(DetectionScore(name="Ensemble AI Score", value=0.73)) assert result_to_ai_probability(result) == pytest.approx(0.73) + def test_prefers_binoculars_calibrated_score(self): + """Binoculars exposes a calibrated probability; the coarse verdict/ + confidence fallback must not be used (it distorts ECE + thresholds).""" + result = AnalysisResult(verdict=Verdict.LIKELY_HUMAN, confidence=90) + result.add_score(DetectionScore(name="Binoculars AI Score", value=0.31)) + assert result_to_ai_probability(result) == pytest.approx(0.31) + + def test_falls_back_only_without_calibrated_score(self): + result = AnalysisResult(verdict=Verdict.LIKELY_AI, confidence=60) + result.add_score(DetectionScore(name="GPT-2 Perplexity", value=42.0)) + # No calibrated primary score -> verdict/confidence fallback. + assert result_to_ai_probability(result) == pytest.approx(0.8) + def test_ai_verdict_maps_above_half(self): result = AnalysisResult(verdict=Verdict.AI_GENERATED, confidence=80) assert result_to_ai_probability(result) == pytest.approx(0.9)