satyamshivam13 · satyamshivam13 · Jul 13, 2026 · Jul 13, 2026 · Jul 13, 2026 · coderabbitai
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,15 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Changed (behaviour — ethics)
+- **The default ensemble verdict is now Binoculars-driven** (`weight_binoculars=1.0`,
+  GPT-2/NLTK/RoBERTa weight 0). The fairness evaluation showed the old
+  GPT-2-weighted blend flagged 71% of non-native English writers; Binoculars-only
+  is 5.5%. GPT-2/NLTK still run and their sub-scores display for transparency, but
+  they no longer drive the verdict. `method_name` is now
+  "Ensemble (Binoculars-weighted)". Raise the GPT-2/NLTK weights only if you accept
+  the fairness cost. Ensemble now loads a second small model (distilgpt2).
+
 ### Added
 - **Per-population fairness evaluation** (`scripts/prepare_fairness_set.py`,
   `scripts/fpr_by_population.py`, `docs/benchmarks/FAIRNESS.md`): false-positive

diff --git a/README.md b/README.md
@@ -47,7 +47,7 @@ This project ships **three independent Streamlit apps** — one per detection en
 |------|-----------|----------------|---------|---------------|--------|--------|
 | **NLTK** | `app.py` | `streamlit run app.py` | Statistical detection via NLTK n-gram language models (Brown corpus). No deep-learning model download. | Quick checks; low-resource machines; default starting point | `<1s` | `<1 GB` |
 | **GPT-2** | `gpt2_app.py` | `streamlit run gpt2_app.py` | Perplexity-based detection using the GPT-2 transformer. | Users wanting a deep-learning signal | `2–5s` | `2–3 GB` |
-| **Ensemble** | `ensemble.py` | `streamlit run ensemble.py` | Weighted fusion of GPT-2 + NLTK signals. **Inherits GPT-2's bias — see the fairness warning below; do not use for decisions about people.** | Experimentation only | `5–10s` | `2–3 GB` |
+| **Ensemble** | `ensemble.py` | `streamlit run ensemble.py` | Verdict is **Binoculars-driven by default** (the fairest signal); GPT-2/NLTK sub-scores are shown but weight 0. | Multi-signal view with a fair default verdict | `5–10s` | `2–3 GB` |
 
 ¹ Per-analysis time after models are loaded. The first run is slower: the NLTK mode builds its n-gram model from the Brown corpus, and the GPT-2/Ensemble modes download model weights on first launch (cached thereafter).
 
@@ -157,8 +157,10 @@ accused. Measured on 785 **human-authored** samples (any flag is a false positiv
 
 - **GPT-2 reproduces [Liang et al. 2023](https://github.com/Weixin-Liang/ChatGPT-Detector-Bias):**
   non-native writers are flagged ~26× more than native writers.
-- **The ensemble is the *worst* for fairness** — 71% of non-native writers — because
-  it inherits GPT-2's bias. Its strong aggregate accuracy hid this.
+- **A GPT-2-weighted ensemble was the *worst*** — 71% of non-native writers — because
+  it inherited GPT-2's bias. **The default ensemble verdict is now Binoculars-driven**
+  (GPT-2/NLTK weight 0), so it inherits Binoculars' fairness instead; the table's
+  "Ensemble" column above reflects the old GPT-2-weighted blend, kept as a warning.
 - **Binoculars is the fairest** (the cross-perplexity ratio cancels the effect),
   but still not perfect: 5.5% for non-native writers vs 0% for native speakers.
 

diff --git a/docs/benchmarks/FAIRNESS.md b/docs/benchmarks/FAIRNESS.md
@@ -22,7 +22,12 @@ plus real human answers across five HC3 domains. Neither corpus is redistributed
 
 ## Results (n=785 human samples; % = real people falsely flagged as AI)
 
-| Population | n | **Binoculars** | GPT-2 | Ensemble |
+The **Ensemble** column below is the **old GPT-2-weighted blend**, kept as the
+cautionary baseline. The *current* default ensemble is Binoculars-weighted, so its
+numbers equal the Binoculars column (re-measured: overall 1.0%, non-native 5.5%,
+every native population 0.0%). See "What this means", point 2.
+
+| Population | n | **Binoculars** (= current default ensemble) | GPT-2 | Ensemble (old GPT-2-weighted) |
 |-----------|--:|---------------:|------:|---------:|
 | **Non-native writers (TOEFL)** | 91 | **5.5%** | 26.4% | **71.4%** |
 | US 8th-grade students | 88 | 0.0% | 1.1% | 17.1% |
@@ -47,12 +52,18 @@ fairest: 1.0% overall, and 5.5% [2.4%, 12.2%] on the hardest population. The
 cross-perplexity *ratio* cancels the "simple, predictable text looks AI-generated"
 effect that drives the bias — which is the entire reason the method exists.
 
-**2. The default ensemble is the WORST for fairness, not the best.** 27% of real
-humans overall, and **71% of non-native English writers**, are flagged. Its
-strong aggregate HC3 accuracy (0.95) *masked* this — the essay populations are
-harder than HC3's mixed answers. **Do not use the ensemble to make decisions
-about people, especially non-native speakers.** The README no longer calls it
-"most robust."
+**2. A GPT-2-weighted ensemble is the WORST for fairness.** 27% of real humans
+overall, and **71% of non-native English writers**, were flagged. Its strong
+aggregate HC3 accuracy (0.95) *masked* this — the essay populations are harder
+than HC3's mixed answers.
+
+**This is why the default ensemble verdict is now Binoculars-driven**
+(`weight_binoculars=1.0`, GPT-2/NLTK weight 0). Re-measured with that default,
+the ensemble collapses onto Binoculars' numbers: **overall 1.0% [0.5%, 2.1%],
+non-native 5.5%, every native-speaker population 0.0%.** The Ensemble column in
+the table above is the *old GPT-2-weighted blend*, kept as the cautionary
+baseline. GPT-2/NLTK sub-scores still display for transparency but no longer
+drive the verdict.
 
 **3. GPT-2 reproduces Liang et al. (2023).** 26% of non-native writers vs ~1% of
 native writers. Non-native English is simpler and more predictable, so it has low

diff --git a/docs/benchmarks/fpr_ensemble.json b/docs/benchmarks/fpr_ensemble.json
@@ -1,99 +1,99 @@
 {
   "analyzer": "ensemble",
-  "overall_fpr": 0.2709,
+  "overall_fpr": 0.0101,
   "overall_wilson95": [
-    0.2392,
-    0.3052
+    0.0049,
+    0.0207
   ],
   "populations": [
     {
       "population": "toefl_nonnative",
       "n": 91,
-      "flagged": 65,
-      "fpr": 0.7143,
-      "wilson95_lo": 0.6143,
-      "wilson95_hi": 0.7969,
+      "flagged": 5,
+      "fpr": 0.0549,
+      "wilson95_lo": 0.0237,
+      "wilson95_hi": 0.1222,
       "ambiguous": false
     },
-    {
-      "population": "hc3_wiki_csai",
-      "n": 60,
-      "flagged": 34,
-      "fpr": 0.5667,
-      "wilson95_lo": 0.441,
-      "wilson95_hi": 0.6843,
-      "ambiguous": false
-    },
-    {
-      "population": "toefl_gpt4_polished",
-      "n": 91,
-      "flagged": 47,
-      "fpr": 0.5165,
-      "wilson95_lo": 0.4152,
-      "wilson95_hi": 0.6164,
-      "ambiguous": true
-    },
     {
       "population": "hc3_open_qa",
       "n": 60,
-      "flagged": 23,
-      "fpr": 0.3833,
-      "wilson95_lo": 0.2709,
-      "wilson95_hi": 0.5098,
+      "flagged": 2,
+      "fpr": 0.0333,
+      "wilson95_lo": 0.0092,
+      "wilson95_hi": 0.1136,
       "ambiguous": false
     },
     {
-      "population": "hc3_finance",
-      "n": 60,
-      "flagged": 15,
-      "fpr": 0.25,
-      "wilson95_lo": 0.1578,
-      "wilson95_hi": 0.3723,
+      "population": "student_us_8th",
+      "n": 88,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0418,
       "ambiguous": false
     },
     {
       "population": "college_admission",
       "n": 70,
-      "flagged": 17,
-      "fpr": 0.2429,
-      "wilson95_lo": 0.1575,
-      "wilson95_hi": 0.355,
-      "ambiguous": false
-    },
-    {
-      "population": "student_us_8th",
-      "n": 88,
-      "flagged": 15,
-      "fpr": 0.1705,
-      "wilson95_lo": 0.1061,
-      "wilson95_hi": 0.2624,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.052,
       "ambiguous": false
     },
     {
       "population": "cs224n_student",
       "n": 145,
-      "flagged": 14,
-      "fpr": 0.0966,
-      "wilson95_lo": 0.0584,
-      "wilson95_hi": 0.1555,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0258,
       "ambiguous": false
     },
+    {
+      "population": "toefl_gpt4_polished",
+      "n": 91,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0405,
+      "ambiguous": true
+    },
     {
       "population": "hc3_reddit_eli5",
       "n": 60,
-      "flagged": 3,
-      "fpr": 0.05,
-      "wilson95_lo": 0.0171,
-      "wilson95_hi": 0.137,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0602,
+      "ambiguous": false
+    },
+    {
+      "population": "hc3_finance",
+      "n": 60,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0602,
       "ambiguous": false
     },
     {
       "population": "hc3_medicine",
       "n": 60,
-      "flagged": 2,
-      "fpr": 0.0333,
-      "wilson95_lo": 0.0092,
-      "wilson95_hi": 0.1136,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0602,
+      "ambiguous": false
+    },
+    {
+      "population": "hc3_wiki_csai",
+      "n": 60,
+      "flagged": 0,
+      "fpr": 0.0,
+      "wilson95_lo": 0.0,
+      "wilson95_hi": 0.0602,
       "ambiguous": false
     }
   ]

diff --git a/ensemble.py b/ensemble.py
@@ -107,9 +107,11 @@
 def load_analyzer() -> EnsembleAnalyzer:
     """Load and cache the Ensemble analyzer."""
     analyzer = EnsembleAnalyzer()
-    # Warm the analyzers that actually contribute to the blend. RoBERTa is
-    # intentionally NOT loaded here: it is disabled (weight 0) and warming it
-    # would trigger a large download and memory use for no contribution.
+    # The default verdict is Binoculars-driven, so warm it. GPT-2/NLTK still run
+    # (their sub-scores are shown for transparency); RoBERTa stays unloaded when
+    # its weight is 0.
+    if analyzer.weights.get("binoculars", 0.0) > 0:
+        _ = analyzer.binoculars_analyzer
     if analyzer.weights.get("roberta", 0.0) > 0:
         _ = analyzer.roberta_analyzer
     _ = analyzer.gpt2_analyzer
@@ -146,18 +148,17 @@ def load_chart_generator() -> ChartGenerator:
     st.markdown("---")
     st.markdown("### ℹ️ About")
     st.markdown("""
-**Ensemble Analyzer** combines two calibrated detection signals:
+**Verdict signal: Binoculars (100%).**
 
-🧠 **GPT-2** (75% weight)
-- Deep perplexity analysis
-- Transformer-based patterns
+🔭 **Binoculars** — cross-perplexity of two models. It alone drives the default
+verdict, because it is the fairest signal (see below).
 
-📊 **NLTK** (25% weight)
-- Statistical n-gram models
-- Linguistic features
+🧠 **GPT-2** / 📊 **NLTK** (0% weight) — sub-scores shown for transparency, but
+they do **not** decide the verdict. A GPT-2-weighted blend flagged **71% of
+non-native English writers** as AI; Binoculars, 5.5%. See
+`docs/benchmarks/FAIRNESS.md`.
 
-⚠️ **Note**: RoBERTa is disabled (weight 0, not loaded) until a fine-tuned
-checkpoint is wired in. See README.
+⚠️ RoBERTa is disabled (untrained head).
 
 **Benchmarks:** see `docs/benchmarks/`
 
@@ -190,7 +191,7 @@ def load_chart_generator() -> ChartGenerator:
 
 ✅ **Transparent** - See how each analyzer votes
 
-✅ **Weighted fusion** - GPT-2 75% / NLTK 25% (configurable)
+✅ **Fair by default** - verdict is Binoculars-driven (GPT-2/NLTK shown but weight 0)
 """)
 
     st.markdown("---")

diff --git a/src/analyzers/ensemble_analyzer.py b/src/analyzers/ensemble_analyzer.py
@@ -49,7 +49,7 @@ class EnsembleAnalyzer(BaseAnalyzer):
 
     def __init__(self):
         super().__init__()
-        self.method_name = "Ensemble (GPT2+NLTK)"
+        self.method_name = "Ensemble (Binoculars-weighted)"
         self.ensemble_config = self.settings.ensemble
 
         # Initialize analyzers
@@ -452,9 +452,12 @@ def _generate_ensemble_explanation(
         )
 
         parts.append(
-            f"\n⚖️ **Weights**: RoBERTa {self.weights['roberta'] * 100:.0f}%, "
+            f"\n⚖️ **Verdict weights**: Binoculars {self.weights['binoculars'] * 100:.0f}%, "
             f"GPT-2 {self.weights['gpt2'] * 100:.0f}%, "
-            f"NLTK {self.weights['nltk'] * 100:.0f}%."
+            f"NLTK {self.weights['nltk'] * 100:.0f}%, "
+            f"RoBERTa {self.weights['roberta'] * 100:.0f}%. "
+            f"GPT-2/NLTK scores are shown for transparency but do not drive the "
+            f"default verdict — see docs/benchmarks/FAIRNESS.md."
         )
 
         # Key metrics

diff --git a/src/config/settings.py b/src/config/settings.py
@@ -190,14 +190,20 @@ class EnsembleConfig:
     nltk_ppl_midpoint: float = 1550.0
     nltk_ppl_slope: float = 0.0015
 
-    # Fusion weights. RoBERTa stays disabled until a fine-tuned checkpoint is
-    # wired in; GPT-2 is the strongest signal and dominates the blend.
-    # Binoculars is available but off by default (it needs a second model); to
-    # enable it, give it a non-zero weight and rebalance so the weights sum to 1.
+    # Fusion weights. The default ensemble VERDICT is driven entirely by
+    # Binoculars (weight 1.0): the per-population fairness evaluation
+    # (docs/benchmarks/FAIRNESS.md) showed that a GPT-2-weighted blend flagged
+    # 71% of non-native English writers as AI, because it inherited GPT-2's
+    # bias. Binoculars' cross-perplexity ratio does not have that bias (5.5% for
+    # the same population), so it is the only signal that should decide the
+    # verdict by default. GPT-2 and NLTK still RUN (their sub-scores are shown
+    # for transparency) but contribute 0 to the verdict; raise their weights
+    # only if you understand the fairness cost. RoBERTa stays disabled (untrained
+    # head). Weights must sum to 1.
     weight_roberta: float = 0.0
-    weight_gpt2: float = 0.75
-    weight_nltk: float = 0.25
-    weight_binoculars: float = 0.0
+    weight_gpt2: float = 0.0
+    weight_nltk: float = 0.0
+    weight_binoculars: float = 1.0
 
     # Verdict thresholds on the fused, calibrated AI-probability.
     ai_threshold: float = 0.70