feat(ensemble): default verdict is Binoculars-driven (fixes the 71% non-native FPR) - #6
Conversation
The per-population fairness evaluation showed the GPT-2-weighted ensemble flagged 71% of non-native English writers as AI (it inherited GPT-2's bias), vs 5.5% for Binoculars. Change the default so the ensemble VERDICT is driven entirely by Binoculars, which is both the most accurate and the fairest signal. - EnsembleConfig default weights: binoculars=1.0, gpt2=0.0, nltk=0.0, roberta=0.0. - GPT-2 and NLTK still run and their sub-scores display for transparency, but contribute 0 to the fused verdict. RoBERTa stays unloaded (weight 0). - method_name -> "Ensemble (Binoculars-weighted)"; explanation and the ensemble app copy updated to state the verdict is Binoculars-only and why. - ensemble.py warms the Binoculars models. Tests updated: mocked ensemble tests now also mock Binoculars (the default runs it); fusion-mechanics tests opt into the legacy GPT-2/NLTK weighting explicitly; new test asserts the fused score equals the Binoculars probability by default. Full fast suite: 236 passed, 85% coverage, lint clean. An end-to-end FPR re-run on the new default is in progress and will be added to docs/benchmarks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 32 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe ensemble’s default verdict is now driven entirely by Binoculars. GPT-2 and NLTK continue to run for transparency, while labels, explanations, startup behavior, tests, and documentation reflect the new weighting. ChangesEnsemble default behavior
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
tests/test_ensemble_weighted_fusion.py (1)
94-99: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winUse the name-based score lookup in this test.
The suite now protects score reordering, but this assertion still reads
combined.scores[0]. Retrieveanalyzer.ENSEMBLE_SCORE_NAMEwithget_score()so the regression remains valid if score order changes.Proposed adjustment
- assert abs(combined.scores[0].value - 0.2) < 1e-6 + primary = combined.get_score(analyzer.ENSEMBLE_SCORE_NAME) + assert primary is not None + assert abs(primary.value - 0.2) < 1e-6🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_ensemble_weighted_fusion.py` around lines 94 - 99, Update the assertion in the weighted fusion test to retrieve the fused score by calling combined.get_score(analyzer.ENSEMBLE_SCORE_NAME) instead of indexing combined.scores[0]. Preserve the existing comparison against 0.2 and tolerance.tests/test_ensemble_analyzer.py (1)
76-83: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAssert the Binoculars signal through
analyze().The new fake is wired correctly, but current coverage mainly exercises
_combine_resultsdirectly. Add an integration assertion thatanalyze()produces an ensemble score equal to the mocked Binoculars probability; otherwise a regression in the forwarding path could go undetected.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_ensemble_analyzer.py` around lines 76 - 83, Extend the ensemble analyzer tests using _configure_mock_analyzers to call analyze() and assert that the resulting ensemble score equals the mocked binoculars_ai probability. Keep the existing _combine_results coverage, while specifically validating that analyze() forwards the Binoculars signal into the final result.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 50: Update the Binoculars setup section in README.md to reflect that
Ensemble now enables Binoculars by default with weight_binoculars=1.0. Remove or
revise statements claiming it is excluded from the default ensemble or requires
a non-zero weight, while preserving accurate usage guidance.
In `@src/analyzers/ensemble_analyzer.py`:
- Around line 455-460: The verdict-weight formatting in analyze() directly
indexes the custom weights map for binoculars, causing legacy maps without that
key to raise KeyError. Replace the binoculars lookup with the existing
defaulting behavior, using weights.get("binoculars", 0.0), while preserving the
current display and fusion behavior.
---
Nitpick comments:
In `@tests/test_ensemble_analyzer.py`:
- Around line 76-83: Extend the ensemble analyzer tests using
_configure_mock_analyzers to call analyze() and assert that the resulting
ensemble score equals the mocked binoculars_ai probability. Keep the existing
_combine_results coverage, while specifically validating that analyze() forwards
the Binoculars signal into the final result.
In `@tests/test_ensemble_weighted_fusion.py`:
- Around line 94-99: Update the assertion in the weighted fusion test to
retrieve the fused score by calling
combined.get_score(analyzer.ENSEMBLE_SCORE_NAME) instead of indexing
combined.scores[0]. Preserve the existing comparison against 0.2 and tolerance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 399262d7-b9e8-4e3e-b59c-70cf403f15e2
📒 Files selected for processing (7)
CHANGELOG.mdREADME.mdensemble.pysrc/analyzers/ensemble_analyzer.pysrc/config/settings.pytests/test_ensemble_analyzer.pytests/test_ensemble_weighted_fusion.py
| | **NLTK** | `app.py` | `streamlit run app.py` | Statistical detection via NLTK n-gram language models (Brown corpus). No deep-learning model download. | Quick checks; low-resource machines; default starting point | `<1s` | `<1 GB` | | ||
| | **GPT-2** | `gpt2_app.py` | `streamlit run gpt2_app.py` | Perplexity-based detection using the GPT-2 transformer. | Users wanting a deep-learning signal | `2–5s` | `2–3 GB` | | ||
| | **Ensemble** | `ensemble.py` | `streamlit run ensemble.py` | Weighted fusion of GPT-2 + NLTK signals. **Inherits GPT-2's bias — see the fairness warning below; do not use for decisions about people.** | Experimentation only | `5–10s` | `2–3 GB` | | ||
| | **Ensemble** | `ensemble.py` | `streamlit run ensemble.py` | Verdict is **Binoculars-driven by default** (the fairest signal); GPT-2/NLTK sub-scores are shown but weight 0. | Multi-signal view with a fair default verdict | `5–10s` | `2–3 GB` | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Update the stale Binoculars setup instructions.
This new row says Binoculars is enabled by default, but the later Binoculars section still says it is not in the default ensemble and must be given a non-zero weight. Update that section to describe the current weight_binoculars=1.0 default.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 50, Update the Binoculars setup section in README.md to
reflect that Ensemble now enables Binoculars by default with
weight_binoculars=1.0. Remove or revise statements claiming it is excluded from
the default ensemble or requires a non-zero weight, while preserving accurate
usage guidance.
| f"\n⚖️ **Verdict weights**: Binoculars {self.weights['binoculars'] * 100:.0f}%, " | ||
| f"GPT-2 {self.weights['gpt2'] * 100:.0f}%, " | ||
| f"NLTK {self.weights['nltk'] * 100:.0f}%." | ||
| f"NLTK {self.weights['nltk'] * 100:.0f}%, " | ||
| f"RoBERTa {self.weights['roberta'] * 100:.0f}%. " | ||
| f"GPT-2/NLTK scores are shown for transparency but do not drive the " | ||
| f"default verdict — see docs/benchmarks/FAIRNESS.md." |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve compatibility with custom weight maps.
tests/test_ensemble_analyzer.py:test_custom_weights uses a legacy map without binoculars. If that map reaches analyze(), this new direct lookup raises KeyError; the broad handler then returns UNCERTAIN even though fusion succeeded. Use .get("binoculars", 0.0) consistently with the rest of the implementation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/analyzers/ensemble_analyzer.py` around lines 455 - 460, The
verdict-weight formatting in analyze() directly indexes the custom weights map
for binoculars, causing legacy maps without that key to raise KeyError. Replace
the binoculars lookup with the existing defaulting behavior, using
weights.get("binoculars", 0.0), while preserving the current display and fusion
behavior.
…lations End-to-end re-run of the per-population FPR with the NEW default ensemble (weight_binoculars=1.0). It collapses onto Binoculars' numbers, as intended: overall FPR 27.1% -> 1.0% [0.5%, 2.1%] non-native 71.4% -> 5.5% native essays 10-24% -> 0.0% (every native population) fpr_ensemble.json now holds the current default's numbers; the FAIRNESS.md table labels the old GPT-2-weighted blend as the cautionary baseline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
✅ Validated end-to-end. Re-ran the per-population FPR with the new default ensemble (785 human samples):
The gating works: the ensemble now inherits Binoculars' fairness. Report updated in |
1 participant
Acts on the fairness finding from #5: the GPT-2-weighted ensemble flagged 71% of non-native English writers as AI. This makes the ensemble's default verdict Binoculars-driven — the signal that is both the most accurate and the fairest.
Change
EnsembleConfigdefault weights: binoculars=1.0, gpt2=0.0, nltk=0.0, roberta=0.0.method_name→ "Ensemble (Binoculars-weighted)"; app copy + narrative explain that the verdict is Binoculars-only and why (link todocs/benchmarks/FAIRNESS.md).distilgpt2(Binoculars' performer).Why keep GPT-2/NLTK running at weight 0?
Transparency — the ensemble's purpose is a multi-signal view. Users still see what GPT-2 and NLTK "thought", clearly marked as non-deciding. Only the verdict is gated to the fair signal.
Tests
Validation in progress
An end-to-end per-population FPR re-run on the new default ensemble is running; expected to drop from 27%/71% to ≈ Binoculars (1% / 5.5%). I'll post the confirmed numbers here and add the report to
docs/benchmarks/before this merges.🤖 Generated with Claude Code
Summary by CodeRabbit