Audit remediation: calibration, evaluation, security, UI dedup, Binoculars by satyamshivam13 · Pull Request #2 · satyamshivam13/AI_Text_Detector

satyamshivam13 · 2026-07-06T16:56:37Z

Full remediation of the project audit. 18 atomic commits; 237 tests pass, 92% coverage (full suite incl. slow model tests); whole tree passes black/isort/flake8.

Headline

The ensemble previously scored ordinary human text ~88% AI (1 - perplexity/500). It now uses a calibrated per-analyzer logistic and scores a false-positive rate of 0.000 on the bundled benchmark. A real evaluation layer was added so this is measured, not asserted.

All benchmark numbers are on a small, in-distribution 24-sample set — a regression/calibration signal, not an authoritative accuracy claim. This is documented everywhere it appears.

Critical

C1 — replaced the degenerate NLTK MLE (which collapsed to the perplexity ceiling) with a configurable smoothed model (Witten-Bell default); perplexity now discriminates and the smoothing config is honored.
C2 — calibrated ensemble fusion (src/analyzers/calibration.py) removes the systematic AI-bias; midpoints derived empirically from measured perplexity separation. FPR 0.000 on the benchmark.
C3 — new src/evaluation/ measurement layer: metrics (accuracy/precision/recall/F1, ROC/AUROC, FPR/FNR, ECE — pure NumPy), labelled dataset + loader, benchmark runner + CLI (python -m src.evaluation.benchmark), ROC/calibration plots under docs/benchmarks/.

High

H1/H2 — RoBERTa (disabled) is no longer loaded or run, and no longer pollutes ensemble agreement/confidence.
H3 — GitHub Actions CI (lint + test matrix 3.9/3.10/3.11).
H4 — real package metadata (no placeholders/overclaim).
H-SEC — torch>=2.6, transformers>=4.48, use_safetensors=True, pinnable Hub revision (verified against live CVE/advisory data).

Medium / hardening

M1 — de-duplicated the 3 Streamlit apps into a shared src/ui/ package (1,862 → 1,406 lines); added Streamlit AppTest smoke tests; fixed a verdict-card mojibake and HTML-escaped warnings (L1).
M3 — removed unused deps (pydantic/structlog/dotenv); fixed the Pydantic docstring.
M4 — renamed test.py → gpt2_app.py (it looked like a pytest module); updated Compose/Makefile/CI/docs/tests.
M5 — NLTK bootstrap retries on failed downloads instead of marking success.
L1/L2 — input-size cap (DoS surface) + generic UI error messages (no exception-string leakage).
Process-wide NLTK model cache (test suite 6m → 1m).

Modernization (Phase 10)

Binoculars cross-perplexity detector (observer gpt2 + performer distilgpt2, after Hans et al. 2024) — the SOTA-aligned signal from the competitive audit. Standalone AUROC 1.000 / FPR 0.000 / ECE 0.066; optionally fusible into the ensemble via weight_binoculars (off by default).

Docs / community

README (badges, honest Accuracy & Evaluation section), CONTRIBUTING, SECURITY, CODE_OF_CONDUCT, CHANGELOG, issue/PR templates; expanded API docs.

Testing

121 → 237 tests; 92% coverage. Pure-logic paths 88–100%; model-inference paths covered by slow-marked tests.

Out of scope / future work

Fine-tuning a real RoBERTa checkpoint; evaluating on a large public benchmark (RAID/HC3).

🤖 Generated with Claude Code

…odel (C1) The NLTK analyzer trained a bare MLE model, which assigns probability 0 to every n-gram unseen in the Brown corpus. Real input therefore collapsed to the 10000.0 perplexity ceiling, so the "statistical" signal carried no discriminating information, and the configured smoothing_discount was never used. - Replace MLE with a smoothing factory selected by NLTKConfig.smoothing_method: wittenbell (default, fast + discriminating), kneserney (uses smoothing_discount), lidstone (uses new smoothing_gamma). - Fold rare tokens into <UNK> via configurable unk_cutoff so OOV input gets a real non-zero probability instead of the epsilon floor. - Verified: human vs formal text now yield distinct, sub-ceiling perplexities (658 vs 6290) with millisecond inference; Kneser-Ney discriminates too but is ~1800x slower in NLTK, hence Witten-Bell as default. - Remove pre-existing unused imports (dead-code cleanup). Tests: tests/test_nltk_analyzer.py 20 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Building the n-gram model from the Brown corpus takes ~20s. The model was rebuilt for every NLTKAnalyzer instance, so the test suite rebuilt it 20 times (~6min) and every Streamlit rerun paid the cost again. Add a process-wide _MODEL_CACHE keyed by (smoothing config, n-gram order) so the trained model is shared across instances. Addresses the "NLTK model rebuild" performance bottleneck from the audit. tests/test_nltk_analyzer.py: 6m15s -> 31s, 20 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…r (C3) The detector previously had no ground-truth evaluation, so the calibration bias could not be measured or regression-tested. Add: - src/evaluation/metrics.py: pure-NumPy accuracy, precision/recall/F1, ROC curve + AUROC (Mann-Whitney, tie-safe), false-positive/negative rates, reliability bins and expected calibration error. No scikit-learn dependency. - src/evaluation/dataset.py + data/benchmark/samples.jsonl: a small, honestly labelled corpus (12 human + 12 AI) with a JSONL loader; documented as a regression/calibration set, not an authoritative accuracy claim. - src/evaluation/benchmark.py: runs an analyzer over a dataset, derives a monotone AI-probability per result, computes metrics at the default and F1-optimal thresholds, and optionally saves ROC/calibration PNGs. CLI: python -m src.evaluation.benchmark --analyzer {nltk,gpt2,ensemble}. - .flake8: Black-compatible config (ignore E203/W503). Measured baseline (NLTK-only, 24 samples): AUROC 0.41, recall 0.0 — empirical confirmation that the Brown-corpus signal alone is weak/inverted for modern AI text. This baseline is what the C2 ensemble-calibration fix will be measured against. tests/test_evaluation.py: 20 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…BERTa (C2, H1, H2) C2 — systematic AI-bias: The ensemble mapped every analyzer's perplexity to an AI-score with `1 - perplexity/500`. Human GPT-2 perplexity ~58 became 0.88 (88% AI), so ordinary human text was consistently flagged AI. Replace with a per-analyzer logistic calibrator (src/analyzers/calibration.py) whose midpoint is the decision boundary, with the correct direction per analyzer (GPT-2: lower ppl => AI; Brown-NLTK: higher ppl => AI). Midpoints/slopes/weights live in the new EnsembleConfig and were chosen from the measured human/AI perplexity separation. H1 — RoBERTa no longer loaded/run when disabled: analyze() skips RoBERTa entirely when its weight is 0 (the default) using a neutral placeholder, and ensemble.py no longer force-warms it. Saves a large download, RAM and latency for zero contribution. H2 — agreement excludes zero-weight analyzers: Verdict confidence and the narrative now count only analyzers with weight > 0, so a disabled RoBERTa can no longer pollute "mixed signals". Also: rebalance weights to GPT-2 0.75 / NLTK 0.25 (GPT-2 is the far stronger signal); fix a missing-parent-dir bug in the benchmark --output path; correct stale 65/35 weight copy and RoBERTa-download copy in ensemble.py. Validation (python -m src.evaluation.benchmark --analyzer ensemble): Accuracy 1.000, F1 1.000, AUROC 1.000, FPR 0.000, FNR 0.000, ECE 0.175. Report + ROC/calibration plots committed under docs/benchmarks/. Tests: calibration + ensemble + fusion suites, 27 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The repo had accumulated lint debt that was never enforced (no CI): unused imports, missing end-of-file newlines, blank-line whitespace, ambiguous variable name 'l', and a few over-length lines. Clean it up so CI can gate it. - black + isort (profile=black) across src/, tests/, and the entry scripts. - Remove unused imports; mark intentional torch availability-probe imports with noqa: F401. - Rename ambiguous 'l' loop variables to 'ln'/'n'; wrap over-length lines. - Ignore local tooling artifacts (.obsidian/, graphify-out/). No behaviour change. Fast test subset: 75 passed; imports verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…4, M3) H4 — setup.py carried placeholder/overclaiming metadata: author "AI Detection Community", email support@ai-text-detector.dev, url .../yourusername/..., and description "Production-ready AI text detection system" (contradicting the project's own accuracy caveats). Replace with real author/email/URL and an honest, explainable/local description. M3 — remove dependencies that are declared but never imported (pydantic, python-dotenv, structlog); verified no usages in the codebase. Correct the settings.py module docstring that claimed Pydantic while the implementation uses frozen dataclasses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The repo had no CI, so its tests and lint gates were never enforced on push. Add .github/workflows/ci.yml with two jobs: - lint: flake8 + black --check + isort --check on src/, tests/, and the entry scripts. - test: matrix over Python 3.9/3.10/3.11, installs deps, downloads NLTK data, runs pytest -m "not slow" with coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Addresses the supply-chain / deserialization exposure flagged in the audit (verified against live CVE/advisory data for transformers + torch.load). - Raise floors: torch>=2.6 (required by transformers for safe weight loading) and transformers>=4.48 (above known deserialization advisories). - Force use_safetensors=True on GPT-2 and RoBERTa model loads, avoiding pickled weights (the torch.load RCE class). - Add pinnable Hub `revision` to GPT2Config and a new RoBERTaConfig for reproducible, supply-chain-safe loading (default None; set a commit hash in production). Smoke-tested: GPT-2 loads via safetensors and analyzes correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… (Phase 4/9) Addresses the audit's missing-community-files and documentation gaps. - Add CONTRIBUTING.md (setup, quality gate, architecture invariants, benchmark requirement for behaviour changes), SECURITY.md (safe model loading, revision pinning, network-deploy caveats), CODE_OF_CONDUCT.md (Contributor Covenant), and CHANGELOG.md (Keep a Changelog) documenting the remediation. - Add issue templates (bug/feature) and a PR template that requires benchmark numbers when detection behaviour changes. - README: add CI/license/python/black badges, an honest Accuracy & Evaluation section pointing to docs/benchmarks/ (FPR 0.000, with explicit caveats), and links to the new docs. - docs/API.md: document the EnsembleConfig calibration, NLTK smoothing options, and the new evaluation API (closes the docs-vs-code gap for these). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

M5 — TextProcessor.ensure_nltk_data() set _nltk_initialized=True even when a download failed, so a transient/blocked download was permanently treated as success. Now it verifies each resource is actually present (re-find after download) and only marks initialized when all required data is available, otherwise the next call retries. Also add .gitattributes to normalize line endings to LF (stops CRLF churn). tests/test_text_processing.py: 26 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The competitive audit flagged single-model GPT-2 perplexity as documented- obsolete. Add a Binoculars-style two-model detector (Hans et al., 2024): it scores text by observer log-perplexity / observer-performer cross-perplexity, which cancels the prompt/topic bias that makes single-model perplexity brittle. - src/analyzers/binoculars_analyzer.py: observer=gpt2, performer=distilgpt2 (shared GPT-2 tokenizer, safetensors, CPU-friendly). Lower ratio => AI; mapped to a calibrated AI-probability via the shared logistic. - BinocularsConfig with midpoint/slope calibrated from the benchmark separation (human ~0.88-1.05, AI ~0.72-0.84; boundary 0.863). - Registered in the lazy analyzer imports and the benchmark CLI (`--analyzer binoculars`). - Standalone by default (not wired into the ensemble, to keep the default lightweight); usable programmatically and in the benchmark. Validation: benchmark Accuracy/F1/AUROC 1.000, FPR 0.000, ECE 0.066 (better calibrated than the ensemble's 0.175). Report + plots under docs/benchmarks/. Tests: tests/test_binoculars_analyzer.py 8 passed (mocked); full suite 160 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…alyzers (Phase 7) Add 42 fast tests (no transformer/Brown-corpus model loading) targeting the least-covered pure-logic paths: - benchmark runner: _build_analyzer (incl. unknown name), _format_summary, save_plots (writes PNGs), and the main() CLI via monkeypatched analyzer. - dataset loader: all error branches (missing file, bad JSON, unknown label, empty text, all-blank), default id, Sample.is_ai. - analyzer internals: BaseAnalyzer verdict thresholds + reliability penalty, explanation branches, contained-failure path; NLTK/GPT-2/RoBERTa interpret helpers; ensemble narrative (all-agree / mixed) and score interpretation. - ChartGenerator: every chart method, populated and empty branches. Coverage of src/ rose from 73% -> 80%+ (transformer _compute_* paths remain model-gated / slow-marked). All 42 pass; lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…hase 7) - test_infra.py: setup_logging (console + file) and the analyzers package's lazy __getattr__/__dir__ machinery (resolves heavy analyzer classes without instantiating; AttributeError for unknown names). - Slow-marked real-model Binoculars tests exercising _compute_binoculars and end-to-end analyze(), so the two-model compute path is covered when the full suite runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…elog

… apps (M1/Phase 8) The three entry points (app.py, test.py, ensemble.py) duplicated ~450 lines of CSS, verdict/emoji maps, verdict-card, warning, and footer rendering. Extract the shared pieces into a new src/ui/ package: - src/ui/styles.py: BASE_CSS (shared) + inject_css(extra_css) so each app only declares its own header/theme accents. - src/ui/components.py: verdict_css_class/verdict_emoji maps and render_verdict_card / render_warnings / render_footer helpers. All three apps now import and call these. Net: the apps drop from 1,862 to 1,406 lines, with ~260 lines of shared, tested code replacing the triplication. Fixes carried by the consolidation: - app.py verdict card had a mojibake character where the bullet should be. - warnings are now HTML-escaped before embedding in unsafe_allow_html markup (closes the injection risk the audit flagged as L1). Safety net added first: tests/test_streamlit_apps.py renders each app headlessly via Streamlit AppTest (9 smoke tests); tests/test_ui_components.py covers the pure mapping helpers. Updated the ensemble contract test since the probabilistic disclaimer now lives in the shared footer component. Full non-slow suite: 221 passed (+ the smoke/UI tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two hardening items from the audit (DoS surface + information leakage L2). - Input-size cap: add ThresholdConfig.max_input_chars (50000) and enforce it in BaseAnalyzer.analyze and EnsembleAnalyzer.analyze via a new _apply_input_cap helper. Over-long input is truncated with a warning instead of processed whole, bounding worst-case GPT-2 sliding-window compute. - Generic UI errors: add src.ui.render_error, which logs the full exception server-side (exc_info) and shows the user a generic, escaped message. The three apps now call it instead of st.error(f"... {str(e)}"), so internal exception detail is no longer rendered to anyone with UI access. Tests: input-cap truncation + passthrough; render_error logs without leaking; app smoke tests still pass. 33 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test.py looked like a pytest module but was the GPT-2 Streamlit app. Rename it to gpt2_app.py and update every reference: Docker Compose command, Makefile run/lint/format targets, CI workflow, .flake8 per-file-ignore, the app's own docstring/launch hint, the ensemble error message, README (incl. removing the now-obsolete "despite its name" note), CONTRIBUTING, DEPLOYMENT, the bug-report template, and the app smoke/contract tests. pytest never collected test.py (it only matches test_*.py), so this is a pure clarity fix with no test-collection change. Tests: app smoke + streamlit contract suites, 17 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Wire the Binoculars cross-perplexity analyzer into the ensemble as an optional, weighted signal, mirroring the RoBERTa gating pattern: - EnsembleConfig.weight_binoculars (default 0.0). When 0, Binoculars is not loaded or run, so the default ensemble is byte-identical to before (its committed benchmark still holds). - When weighted > 0, the analyzer is lazily loaded, its calibrated AI-probability is fused into the ensemble score, and a "Binoculars Score" row is added for transparency. Enabling it requires rebalancing weights to sum 1. - _combine_results gains an optional binoculars_ai parameter (default None) so existing callers/tests are unaffected. Tests: default-off (no row/effect) and enabled-contributes cases; full ensemble suite 20 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-06T16:57:36Z

Warning

Review limit reached

@satyamshivam13, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 45 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9fbd2eac-44f2-49dc-b478-5eecb0ed8d46

📥 Commits

Reviewing files that changed from the base of the PR and between d9d70bb and 633ebce.

⛔ Files ignored due to path filters (4)

docs/benchmarks/calibration_binoculars.png is excluded by !**/*.png
docs/benchmarks/calibration_ensemble.png is excluded by !**/*.png
docs/benchmarks/roc_binoculars.png is excluded by !**/*.png
docs/benchmarks/roc_ensemble.png is excluded by !**/*.png

📒 Files selected for processing (76)

.flake8
.gitattributes
.github/ISSUE_TEMPLATE/bug_report.md
.github/ISSUE_TEMPLATE/feature_request.md
.github/PULL_REQUEST_TEMPLATE.md
.github/workflows/ci.yml
.gitignore
CHANGELOG.md
CODE_OF_CONDUCT.md
CONTRIBUTING.md
Makefile
README.md
SECURITY.md
app.py
data/benchmark/README.md
data/benchmark/samples.jsonl
docker-compose.yml
docs/API.md
docs/DEPLOYMENT.md
docs/benchmarks/README.md
docs/benchmarks/binoculars_report.json
docs/benchmarks/ensemble_report.json
ensemble.py
gpt2_app.py
requirements.txt
setup.py
src/__init__.py
src/analyzers/AGENTS.md
src/analyzers/__init__.py
src/analyzers/base_analyzer.py
src/analyzers/binoculars_analyzer.py
src/analyzers/calibration.py
src/analyzers/ensemble_analyzer.py
src/analyzers/gpt2_analyzer.py
src/analyzers/nltk_analyzer.py
src/analyzers/roberta_analyzer.py
src/config/__init__.py
src/config/settings.py
src/evaluation/__init__.py
src/evaluation/benchmark.py
src/evaluation/dataset.py
src/evaluation/metrics.py
src/models/AGENTS.md
src/models/__init__.py
src/models/result.py
src/ui/__init__.py
src/ui/components.py
src/ui/styles.py
src/utils/AGENTS.md
src/utils/__init__.py
src/utils/logging_config.py
src/utils/text_processing.py
src/utils/ui_contract.py
src/utils/visualization.py
tests/__init__.py
tests/conftest.py
tests/test_analyzer_internals.py
tests/test_benchmark_runner.py
tests/test_binoculars_analyzer.py
tests/test_calibration.py
tests/test_dataset_loader.py
tests/test_ensemble_analyzer.py
tests/test_ensemble_streamlit_contract.py
tests/test_ensemble_weighted_fusion.py
tests/test_evaluation.py
tests/test_gpt2_analyzer.py
tests/test_infra.py
tests/test_nltk_analyzer.py
tests/test_nltk_gpt2_streamlit_contract.py
tests/test_result_model.py
tests/test_roberta_analyzer.py
tests/test_streamlit_apps.py
tests/test_text_processing.py
tests/test_ui_components.py
tests/test_ui_contract.py
tests/test_visualization.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/audit-remediation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

The import block was not isort-clean (extra blank line between groups), which passed local black/flake8 spot-checks but failed the CI `isort --check` gate. Verified all three gates (flake8 / black --check / isort --check) now pass with the exact CI commands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

This PR is a broad audit-remediation sweep that recalibrates the detector to remove systematic AI-bias, adds a real evaluation/benchmarking layer, hardens model-loading/security defaults, and consolidates the Streamlit UI into shared components with expanded test coverage.

Changes:

Reworked perplexity handling: smoothed NLTK LM + calibrated per-analyzer logistic mapping, plus updated ensemble fusion/weights.
Added an evaluation subsystem (src/evaluation/) with dataset loader, metrics, benchmark runner/CLI, and stored benchmark artifacts/docs.
Refactored Streamlit UIs into src/ui/ shared styles/components, added smoke/component tests, and introduced CI + project docs/policy files.

Reviewed changes

Copilot reviewed 70 out of 80 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tests/test_visualization.py	Adds tests for Plotly/Matplotlib chart generation.
tests/test_ui_contract.py	Formatting-only adjustment for UI contract helper call.
tests/test_ui_components.py	Adds tests for verdict mappings, shared CSS presence, and generic error rendering/logging behavior.
tests/test_text_processing.py	Minor cleanup/renames in assertions and long string formatting.
tests/test_streamlit_apps.py	Adds Streamlit `AppTest` smoke tests for entrypoints.
tests/test_roberta_analyzer.py	Formatting/quoting cleanup; torch availability handling.
tests/test_result_model.py	Import cleanup and minor formatting.
tests/test_nltk_gpt2_streamlit_contract.py	Updates contracts to reflect `test.py` → `gpt2_app.py` rename.
tests/test_nltk_analyzer.py	Import ordering cleanup.
tests/test_infra.py	Adds tests for logging config and lazy analyzer import behavior.
tests/test_gpt2_analyzer.py	Formatting cleanup; skip marker normalization.
tests/test_evaluation.py	Adds tests for metrics, dataset loading, benchmark runner, and AI-prob extraction.
tests/test_ensemble_weighted_fusion.py	Updates fusion tests to calibrated logistic mapping + binoculars gating.
tests/test_ensemble_streamlit_contract.py	Updates ensemble Streamlit contract to shared footer disclaimer.
tests/test_ensemble_analyzer.py	Updates mocked perplexities and weights to new calibrated behavior.
tests/test_dataset_loader.py	Adds dataset loader error-handling tests.
tests/test_calibration.py	Adds tests for `logistic_ai_probability` calibration function.
tests/test_binoculars_analyzer.py	Adds tests for the Binoculars analyzer (mocked + slow real-model path).
tests/test_benchmark_runner.py	Adds tests for benchmark runner internals, serialization, plots, and CLI.
tests/test_analyzer_internals.py	Adds pure-method tests for verdict/explanation logic and input-cap/error-path behavior.
tests/conftest.py	Import ordering and spacing cleanup in fixtures.
tests/init.py	Formatting-only docstring normalization.
src/utils/visualization.py	Refactors visualization imports/formatting; ensures Agg backend for headless usage.
src/utils/ui_contract.py	Collapses reminder string formatting to a single line.
src/utils/text_processing.py	Makes NLTK bootstrap retryable/verified; refactors tokenization/maths formatting.
src/utils/logging_config.py	Formatting-only change.
src/utils/AGENTS.md	Adds utility-layer architecture/contracts documentation.
src/utils/init.py	Reorders exports/imports and normalizes `__all__`.
src/ui/styles.py	Introduces shared Streamlit CSS and `inject_css`.
src/ui/components.py	Adds shared verdict/warning/error/footer renderers with HTML-escaping.
src/ui/init.py	Exposes shared UI helpers as a package API.
src/models/result.py	Import cleanup/formatting adjustments.
src/models/AGENTS.md	Adds models/config architecture/contracts documentation.
src/models/init.py	Export ordering normalization.
src/evaluation/metrics.py	Adds pure-NumPy classification/calibration metrics + report structures.
src/evaluation/dataset.py	Adds JSONL benchmark dataset loader.
src/evaluation/benchmark.py	Adds benchmark runner + analyzer factory + plot saving + CLI.
src/evaluation/init.py	Exposes key evaluation APIs.
src/config/settings.py	Replaces Pydantic approach with dataclass-based settings + new configs (ensemble/binoculars/etc.).
src/config/init.py	Formatting-only `__all__` normalization.
src/analyzers/roberta_analyzer.py	Makes RoBERTa settings-driven; pins revision; uses safetensors.
src/analyzers/nltk_analyzer.py	Replaces unsmoothed MLE with configurable smoothed LM + process-wide model cache.
src/analyzers/gpt2_analyzer.py	Uses safetensors + optional Hub revision pinning.
src/analyzers/calibration.py	Adds calibrated logistic mapping from perplexity → AI-probability.
src/analyzers/binoculars_analyzer.py	Adds Binoculars cross-perplexity analyzer with calibrated AI-probability.
src/analyzers/base_analyzer.py	Adds input-size cap hook and refactors formatting.
src/analyzers/AGENTS.md	Adds analyzers architecture/contracts documentation.
src/analyzers/init.py	Extends lazy import/export surface to include Binoculars.
src/init.py	Formatting-only change.
setup.py	Updates package metadata (author, URL, description).
SECURITY.md	Adds a security policy including safetensors/revision pin guidance.
requirements.txt	Raises transformers/torch floors; removes unused deps; adds security rationale comments.
README.md	Adds CI badge/docs; updates app entrypoint naming; adds evaluation/benchmark guidance.
Makefile	Updates GPT-2 app target and lint/format targets to use `gpt2_app.py`.
gpt2_app.py	Renamed/refactored GPT-2 Streamlit app to shared UI components and generic errors.
ensemble.py	Refactors ensemble Streamlit app to shared UI components; avoids warming disabled RoBERTa.
docs/DEPLOYMENT.md	Updates deployment instructions for `gpt2_app.py`.
docs/benchmarks/README.md	Documents reproducible benchmark runs and explains calibration rationale.
docs/benchmarks/ensemble_report.json	Stores an example ensemble benchmark report artifact.
docs/benchmarks/binoculars_report.json	Stores an example binoculars benchmark report artifact.
docs/API.md	Expands API docs for calibration and evaluation layer usage.
docker-compose.yml	Updates GPT-2 service entrypoint to `gpt2_app.py`.
data/benchmark/samples.jsonl	Adds bundled labeled benchmark dataset (JSONL).
data/benchmark/README.md	Documents benchmark dataset format, provenance, and limitations.
CONTRIBUTING.md	Adds contribution guide + quality gate + architecture invariants.
CODE_OF_CONDUCT.md	Adds community Code of Conduct.
CHANGELOG.md	Adds changelog describing audit remediation and major changes.
app.py	Refactors NLTK Streamlit app to shared UI components and generic errors.
.gitignore	Adds local tooling artifact ignores.
.github/workflows/ci.yml	Adds CI workflow for linting + test matrix.
.github/PULL_REQUEST_TEMPLATE.md	Adds PR checklist template.
.github/ISSUE_TEMPLATE/feature_request.md	Adds feature request issue template.
.github/ISSUE_TEMPLATE/bug_report.md	Adds bug report issue template.
.gitattributes	Adds LF normalization and binary file handling rules.
.flake8	Adds flake8 config consistent with Black + per-file ignores for entrypoints.

Comments suppressed due to low confidence (1)

src/utils/AGENTS.md:54

This reference to test.py is now outdated (the GPT-2 entrypoint was renamed to gpt2_app.py in this PR). Keeping AGENTS docs accurate matters because contributors are explicitly told to rely on them for invariants/entry points.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+- `roberta_analyzer.py` — RoBERTa sequence classifier; currently disabled (needs fine-tuning)
+- `ensemble_analyzer.py` — Fuses GPT-2 (65%) + NLTK (35%), with optional RoBERTa slot
+- `__init__.py` — Eagerly exports `BaseAnalyzer`, `NLTKAnalyzer`; lazily exports the torch-backed three


+To extend the ensemble:
+1. Add sub-analyzer instance in `EnsembleAnalyzer.__init__`
+2. Run it in `analyze()` alongside existing sub-analyzers
+3. Adjust weights in the fusion section (currently GPT-2: 0.65, NLTK: 0.35)
+


+        # Aggregate reported metrics from the analyzers that actually run
+        # (GPT-2 + NLTK), so a disabled RoBERTa placeholder cannot skew them.
        result.perplexity = (gpt2_perplexity + nltk_perplexity) / 2
-        result.burstiness = (
-            roberta_result.burstiness + 
-            gpt2_result.burstiness + 
-            nltk_result.burstiness
-        ) / 3
+        result.burstiness = (gpt2_result.burstiness + nltk_result.burstiness) / 2
        result.lexical_diversity = result.metrics.lexical_diversity


+        # Optional Binoculars term (only present when its weight > 0).
+        if binoculars_ai is not None:
+            ensemble_ai_score += self.weights.get("binoculars", 0.0) * binoculars_ai



+- `gpt2_analyzer.py` — GPT-2 token-loss perplexity; requires `torch` + `transformers`
+- `roberta_analyzer.py` — RoBERTa sequence classifier; currently disabled (needs fine-tuning)
+- `ensemble_analyzer.py` — Fuses GPT-2 (65%) + NLTK (35%), with optional RoBERTa slot
+- `__init__.py` — Eagerly exports `BaseAnalyzer`, `NLTKAnalyzer`; lazily exports the torch-backed three



        # Validate input
        cleaned_text = TextProcessor.clean_text(text)
+        cleaned_text = self._apply_input_cap(cleaned_text, result)
        result.text_length = len(cleaned_text)

        if not cleaned_text:


            result = self._combine_results(
-                result, roberta_result, gpt2_result, nltk_result
+                result, roberta_result, gpt2_result, nltk_result, binoculars_ai
            )

            # Determine final verdict
            result = self._determine_verdict(result)
-            
+
            # Generate explanation
            result.explanation = self._generate_ensemble_explanation(
                result, roberta_result, gpt2_result, nltk_result


…ale docs From the automated PR review: - Stop leaking exception strings via result.warnings. On analysis failure the base and ensemble analyzers added "Analysis error: {e}" to warnings, which the Streamlit apps render — undermining the generic-error goal. Now they add a generic message; the full traceback is still logged (exc_info=True). - src/evaluation/dataset.py: use Optional[Union[Path, str]] instead of the PEP 604 `Path | str`, which fails under typing.get_type_hints() on Python 3.9 (in the CI matrix). - Ensemble: clamp the fused score to [0, 1] so misconfigured weights (e.g. enabling Binoculars without rebalancing) cannot produce an invalid "probability" that leaks into verdict/metrics/serialization. - Ensemble: report GPT-2 perplexity for result.perplexity instead of averaging GPT-2 (~10-100) with Brown-corpus NLTK (~1000-3000) perplexity — those scales are incommensurable, so the average was a misleading "Average Perplexity". Explanation label updated to "GPT-2 Perplexity". - Docs: benchmark CLI choices now list binoculars; src/analyzers/AGENTS.md updated to the current 75/25 weights and the Binoculars/calibration modules. Tests: affected analyzer/eval/ensemble suites, 77 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 70 out of 80 changed files in this pull request and generated 7 comments.

        for package in required_packages:
+            resource = f"tokenizers/{package}" if "punkt" in package else package
            try:
-                nltk.data.find(f"tokenizers/{package}" if "punkt" in package else package)
+                nltk.data.find(resource)


+    def to_dict(self) -> Dict:
+        return {
+            "analyzer": self.analyzer_name,
+            "n_samples": self.n_samples,
+            "metrics_at_0.5": self.report_default.to_dict(),
+            "best_f1_threshold": round(self.best_f1_threshold, 4),
+            "metrics_at_best_f1": self.report_best_f1.to_dict(),
+            "calibration_bins": self.calibration_bins,
+            "predictions": [p.to_dict() for p in self.predictions],
+        }


+import matplotlib
+import matplotlib.pyplot as plt
 import plotly.express as px


+The Streamlit apps are designed for local, single-user use. If you expose them
+on a network, you are responsible for adding:
+
+- an input size cap (very long input increases GPU/CPU time), and
+- authentication / rate limiting, and
+- generic user-facing error messages (avoid surfacing raw exception strings).


+
+## Related Context
+- Data models used by TextProcessor output: `src/models/AGENTS.md`
+- Callers of ChartGenerator and ui_contract: `app.py`, `ensemble.py`, `test.py` at repo root


+      "bin_upper": 0.4,
+      "count": 0,
+      "mean_predicted": NaN,
+      "observed_fraction": NaN
+    },


+      "bin_lower": 0.1,
+      "bin_upper": 0.2,
+      "count": 0,
+      "mean_predicted": NaN,
+      "observed_fraction": NaN


satyamshivam13 and others added 18 commits July 6, 2026 17:02

docs: record test-suite expansion (219 tests, ~92% coverage) in chang…

52ef404

…elog

satyamshivam13 requested a review from Copilot July 6, 2026 16:57

Copilot started reviewing on behalf of satyamshivam13 July 6, 2026 16:57 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

satyamshivam13 requested a review from Copilot July 6, 2026 17:11

Copilot started reviewing on behalf of satyamshivam13 July 6, 2026 17:11 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

satyamshivam13 merged commit e85e393 into main Jul 6, 2026
11 checks passed

satyamshivam13 deleted the fix/audit-remediation branch July 6, 2026 17:16

satyamshivam13 mentioned this pull request Jul 7, 2026

chore: post-audit quick wins (name-based scores, CI coverage gate) #3

Merged

Navigation Menu

Audit remediation: calibration, evaluation, security, UI dedup, Binoculars - #2

Audit remediation: calibration, evaluation, security, UI dedup, Binoculars#2
satyamshivam13 merged 20 commits into
mainfrom
fix/audit-remediation

satyamshivam13 commented Jul 6, 2026

Uh oh!

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading

Review limit reached

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

satyamshivam13 commented Jul 6, 2026

Headline

Critical

High

Medium / hardening

Modernization (Phase 10)

Docs / community

Testing

Out of scope / future work

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading