Audit remediation: calibration, evaluation, security, UI dedup, Binoculars

.flake8

@@ -0,0 +1,20 @@
+[flake8]
+max-line-length = 100
+# E203 (whitespace before ':') and W503 (line break before binary operator)
+# conflict with Black's formatting; the Black docs recommend ignoring them.
+extend-ignore = E203, W503
+exclude =
+    .git,
+    __pycache__,
+    venv,
+    .venv,
+    build,
+    dist,
+    *.egg-info
+per-file-ignores =
+    # Streamlit entry points must insert src/ on sys.path before importing
+    # src.* (a project-wide invariant documented in AGENTS.md), which trips
+    # E402 (module import not at top of file).
+    app.py:E402
+    gpt2_app.py:E402
+    ensemble.py:E402

.gitattributes

@@ -0,0 +1,10 @@
+# Normalize line endings to LF in the repository; check out native on Windows.
+* text=auto eol=lf
+
+# Binary assets — never normalize.
+*.png binary
+*.jpg binary
+*.jpeg binary
+*.gif binary
+*.safetensors binary
+*.bin binary

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,30 @@
+---
+name: Bug report
+about: Report a problem with detection, the apps, or the library
+title: "[Bug] "
+labels: bug
+---
+
+## Description
+A clear description of the bug.
+
+## To reproduce
+Steps or a minimal code snippet:
+
+```python
+from src.analyzers.nltk_analyzer import NLTKAnalyzer
+result = NLTKAnalyzer().analyze("...")
+```
+
+## Expected vs actual
+- Expected:
+- Actual (include the verdict/confidence/warnings if relevant):
+
+## Environment
+- OS:
+- Python version:
+- Analyzer/app (`app.py` / `gpt2_app.py` / `ensemble.py` / library):
+- Relevant package versions (`pip show torch transformers nltk streamlit`):
+
+## Additional context
+Logs, screenshots, or sample input (avoid sensitive text).

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an improvement (new analyzer, metric, UX, etc.)
+title: "[Feature] "
+labels: enhancement
+---
+
+## Problem / motivation
+What are you trying to do, and why is it hard today?
+
+## Proposed solution
+What you'd like to see. If it changes detection behaviour, note how it should be
+validated (e.g. benchmark metrics that must not regress).
+
+## Alternatives considered
+Other approaches you thought about.
+
+## Does it fit the project's mission?
+This project favours transparent, explainable, local detection. How does the
+request align with that?

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,19 @@
+## Summary
+What does this PR change and why?
+
+## Related
+Closes # (issue) / addresses audit finding (e.g. C2, H1).
+
+## Validation
+- [ ] `pytest tests/ -m "not slow"` passes
+- [ ] `flake8` / `black --check` / `isort --check` pass
+- [ ] If detection behaviour changed, benchmark re-run and numbers included
+      below (false-positive rate on human text must not regress):
+
+```
+python -m src.evaluation.benchmark --analyzer ensemble
+# paste Accuracy / F1 / AUROC / FPR / FNR / ECE here
+```
+
+## Notes for reviewers
+Anything non-obvious, trade-offs, or follow-ups.

.github/workflows/ci.yml

@@ -0,0 +1,54 @@
+name: CI
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+    branches: ["**"]
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    name: Lint & format
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install lint tooling
+        run: |
+          python -m pip install --upgrade pip
+          pip install black flake8 isort
+      - name: flake8
+        run: flake8 src/ tests/ app.py gpt2_app.py ensemble.py --max-line-length=100
+      - name: black --check
+        run: black --check --line-length=100 src/ tests/ app.py gpt2_app.py ensemble.py
+      - name: isort --check
+        run: isort --check-only --profile=black src/ tests/ app.py gpt2_app.py ensemble.py
+
+  test:
+    name: Tests (py${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt -r requirements-dev.txt
+      - name: Download NLTK data
+        run: python -c "import nltk; nltk.download(['punkt','punkt_tab','stopwords','brown','averaged_perceptron_tagger'])"
+      - name: Run tests (excluding slow model tests)
+        run: python -m pytest tests/ -m "not slow" -q --cov=src --cov-report=term-missing

.gitignore

@@ -53,4 +53,7 @@ Thumbs.db
 
 # Distribution
 *.tar.gz
-*.whl
+*.whl
+# Local tooling artifacts (not part of the repo)
+.obsidian/
+graphify-out/

CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project aims
+to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+Remediation of the project audit. Highlights: the ensemble no longer flags
+ordinary human text as AI, the statistical model is now smoothed and
+discriminating, and there is a real evaluation layer.
+
+### Added
+- **Binoculars analyzer** (`src/analyzers/binoculars_analyzer.py`): zero-shot
+  cross-perplexity detection (observer `gpt2` + performer `distilgpt2`) after
+  Hans et al., 2024 — the modern, prompt-robust signal recommended by the audit.
+  Available standalone and via the benchmark (`--analyzer binoculars`);
+  benchmark AUROC 1.000, FPR 0.000, ECE 0.066. Can also be fused into the
+  ensemble via `EnsembleConfig.weight_binoculars` (off by default; loaded only
+  when weighted, mirroring the RoBERTa gating).
+- **Evaluation layer** (`src/evaluation/`): metrics (accuracy, precision,
+  recall, F1, ROC/AUROC, false-positive/negative rates, expected calibration
+  error), a labelled benchmark dataset + loader, and a benchmark runner with a
+  CLI (`python -m src.evaluation.benchmark`).
+- **Perplexity calibration** (`src/analyzers/calibration.py`): per-analyzer
+  logistic mapping from perplexity to a calibrated AI-probability.
+- **Benchmark results and plots** under `docs/benchmarks/`.
+- **CI pipeline** (`.github/workflows/ci.yml`): lint + test matrix (3.9–3.11).
+- Community health files: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`,
+  `SECURITY.md`, issue/PR templates.
+- Configurable NLTK smoothing (`smoothing_method`) and a process-wide model
+  cache.
+- Substantially expanded test suite (121 → 219 tests; ~92% coverage of `src/`
+  with the full suite), covering the evaluation layer, calibration, dataset
+  loader, visualization, analyzer verdict/explanation logic, and the Binoculars
+  compute path.
+- Pinnable Hub `revision` for GPT-2/RoBERTa loading (`GPT2Config`,
+  `RoBERTaConfig`).
+
+### Changed
+- **Renamed the GPT-2 Streamlit app `test.py` → `gpt2_app.py`** so it no longer
+  looks like a pytest module. Updated Docker Compose, Makefile, CI, docs, and
+  the input-size cap / generic UI error handling below.
+- **Ensemble fusion is now calibrated.** Replaced `1 - perplexity/500` (which
+  scored human text ~88% AI) with a per-analyzer logistic whose midpoint is the
+  decision boundary. On the bundled benchmark, false-positive rate dropped to
+  0.000. Weights rebalanced to GPT-2 0.75 / NLTK 0.25.
+- **NLTK model** uses Witten-Bell interpolation (configurable) instead of an
+  unsmoothed MLE that collapsed to the perplexity ceiling.
+- Dependency floors raised to patched releases (`torch>=2.6`,
+  `transformers>=4.48`); model weights loaded with `use_safetensors=True`.
+- Corrected package metadata and removed unused dependencies.
+
+### Fixed
+- RoBERTa (disabled by default) is no longer downloaded or run, and no longer
+  pollutes ensemble agreement/confidence.
+- Whole codebase now passes `black`, `isort`, and `flake8`.
+- Benchmark `--output` no longer fails when the parent directory is missing.
+
+### Security
+- Safe (non-pickle) model loading via safetensors; patched dependency floors;
+  reproducible Hub revision pinning. See `SECURITY.md`.

CODE_OF_CONDUCT.md

@@ -0,0 +1,39 @@
+# Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and maintainers pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes
+
+Examples of unacceptable behavior:
+
+- The use of sexualized language or imagery, and unwelcome sexual attention
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without explicit permission
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the project maintainer at **shivamsatyam35@gmail.com**. All
+complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1.
+
+[homepage]: https://www.contributor-covenant.org

CONTRIBUTING.md

@@ -0,0 +1,68 @@
+# Contributing to AI Text Detector
+
+Thanks for your interest in improving this project. It aims to be a
+**transparent, explainable, local** AI-text-likelihood toolkit — contributions
+that strengthen that mission (better calibration, clearer explanations, honest
+evaluation) are especially welcome.
+
+## Development setup
+
+```bash
+git clone https://github.com/satyamshivam13/AI_Text_Detector.git
+cd AI_Text_Detector
+python -m venv venv
+# Linux/macOS: source venv/bin/activate
+# Windows PowerShell: .\venv\Scripts\Activate.ps1
+pip install -r requirements.txt -r requirements-dev.txt
+python -c "import nltk; nltk.download(['punkt','punkt_tab','stopwords','brown','averaged_perceptron_tagger'])"
+```
+
+## Quality gate (run before every PR)
+
+These mirror the CI checks in `.github/workflows/ci.yml`:
+
+```bash
+python -m pytest tests/ -m "not slow" -q          # tests
+python -m flake8 src/ tests/ app.py gpt2_app.py ensemble.py --max-line-length=100
+python -m black --check --line-length=100 src/ tests/ app.py gpt2_app.py ensemble.py
+python -m isort --check-only --profile=black src/ tests/ app.py gpt2_app.py ensemble.py
+```
+
+Auto-format with `python -m black ... --line-length=100` and
+`python -m isort ... --profile=black` (no `--check`).
+
+## Architecture invariants
+
+Before changing code, read the local `AGENTS.md` in the subdirectory you are
+touching. Key invariants:
+
+- All analysis flows through `analyzer.analyze(text) -> AnalysisResult`. Never
+  call `_perform_analysis` directly.
+- `AnalysisResult`, `TextMetrics`, `DetectionScore` are plain dataclasses.
+- `get_settings()` is the single cached settings entry point; never instantiate
+  `Settings()` directly.
+- Subclasses use `self.thresholds` — never hardcode threshold values.
+- Transformer analyzers are lazily imported so `app.py` runs without `torch`.
+- On any analysis failure: set `Verdict.UNCERTAIN`, zero confidence, append a
+  warning — never let exceptions reach the UI.
+
+## Changing detection behaviour
+
+If you change thresholds, smoothing, calibration, or fusion weights, **re-run
+the benchmark and include the numbers** in your PR:
+
+```bash
+python -m src.evaluation.benchmark --analyzer ensemble
+```
+
+New calibration must not regress the false-positive rate on human text.
+
+## Commit and PR style
+
+- Small, focused commits with a clear subject line (`area: summary`).
+- Reference the audit finding or issue where relevant.
+- Describe what you changed, why, and how you validated it.
+
+## Reporting bugs / requesting features
+
+Use the issue templates under `.github/ISSUE_TEMPLATE/`.

Makefile

@@ -24,7 +24,7 @@ run-nltk: ## Run NLTK-based detector
 	PYTHONPATH=src $(STREAMLIT) run app.py
 
 run-gpt2: ## Run GPT-2-based detector
-	PYTHONPATH=src $(STREAMLIT) run test.py
+	PYTHONPATH=src $(STREAMLIT) run gpt2_app.py
 
 run-ensemble: ## Run Ensemble detector (RoBERTa+GPT2+NLTK)
 	PYTHONPATH=src $(STREAMLIT) run ensemble.py
@@ -33,13 +33,13 @@ test: ## Run tests with coverage
 	PYTHONPATH=src $(PYTHON) -m pytest tests/ -v --cov=src --cov-report=html --cov-report=term-missing
 
 lint: ## Run linters
-	flake8 src/ tests/ app.py test.py ensemble.py --max-line-length=100
+	flake8 src/ tests/ app.py gpt2_app.py ensemble.py --max-line-length=100
 	mypy src/ --ignore-missing-imports
 	pylint src/ --disable=C0114,C0115,C0116
 
 format: ## Format code
-	black src/ tests/ app.py test.py ensemble.py --line-length=100
-	isort src/ tests/ app.py test.py ensemble.py --profile=black
+	black src/ tests/ app.py gpt2_app.py ensemble.py --line-length=100
+	isort src/ tests/ app.py gpt2_app.py ensemble.py --profile=black
 
 clean: ## Clean build artifacts
 	find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true