From cd011a957f8c47e293a58378b9148e4dcd850937 Mon Sep 17 00:00:00 2001 From: Satyam Shivam Date: Tue, 7 Jul 2026 09:00:50 +0530 Subject: [PATCH 1/2] docs: refresh codebase map to post-remediation state Regenerate .planning/codebase/ (7 docs) to reflect main after the audit- remediation merge (PR #2): gpt2_app.py rename, new src/ui/ and src/evaluation/ packages, calibration.py + binoculars_analyzer.py, EnsembleConfig-driven calibrated fusion, safetensors/patched-dep floors, CI, ~231 tests at ~92% coverage, and the current remaining concerns (large-benchmark eval, revision pinning, scores[0] ordering, numpy<2, py3.9 Docker base, CI mypy/coverage gates). Co-Authored-By: Claude Opus 4.8 --- .planning/codebase/ARCHITECTURE.md | 171 ++++++++---------------- .planning/codebase/CONCERNS.md | 208 ++++++++--------------------- .planning/codebase/CONVENTIONS.md | 159 ++++++++-------------- .planning/codebase/INTEGRATIONS.md | 130 ++++++++---------- .planning/codebase/STACK.md | 81 ++++++----- .planning/codebase/STRUCTURE.md | 195 ++++++++------------------- .planning/codebase/TESTING.md | 193 ++++++++------------------ 7 files changed, 384 insertions(+), 753 deletions(-) diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md index 6ec5f35..8db9c4d 100644 --- a/.planning/codebase/ARCHITECTURE.md +++ b/.planning/codebase/ARCHITECTURE.md @@ -1,136 +1,73 @@ # Architecture -**Analysis Date:** 2026-04-02 +**Analysis Date:** 2026-07-06 ## Pattern Overview -**Overall:** Layered monolith with a **Template Method** analyzer hierarchy, optional **ensemble composition**, and **Streamlit scripts as thin presentation shells**. - -**Key Characteristics:** -- Core detection logic lives under `src/` as importable packages; runnable UIs prepend `src` to `sys.path` then import `src.*` explicitly. -- `BaseAnalyzer` centralizes validation, metric computation, verdict scoring, explanations, and timing; concrete analyzers only implement `_perform_analysis`. -- `EnsembleAnalyzer` is a special case: it **overrides** `analyze()` to run multiple backends and fuse outputs instead of using the base `_perform_analysis` hook alone. -- Shared preprocessing and NLTK bootstrap live in `TextProcessor` (`src/utils/text_processing.py`); charts in `ChartGenerator` (`src/utils/visualization.py`). +Layered Python package (`src/`) behind three thin Streamlit UIs. Core detection +uses the **Template Method** pattern: `BaseAnalyzer.analyze()` is concrete and +orchestrates the pipeline; subclasses implement only `_perform_analysis()`. +`EnsembleAnalyzer` is the exception — it **overrides `analyze()`** to run and fuse +multiple sub-analyzers. ## Layers -**Presentation (Streamlit):** -- Purpose: Page config, layout, CSS, widgets, call into analyzers and charts, display `AnalysisResult`. -- Location: repository root — `app.py`, `ensemble.py`, `test.py` (GPT-2 UI; filename is legacy). -- Contains: Procedural Streamlit code, `@st.cache_resource` loaders, verdict-to-CSS mapping. -- Depends on: `src.analyzers.*`, `src.config.settings`, `src.utils.logging_config`, `src.utils.visualization`. -- Used by: End users via `streamlit run .py`. - -**Analysis (domain services):** -- Purpose: Turn raw text into `AnalysisResult` (verdict, scores, metrics, warnings). -- Location: `src/analyzers/` -- Contains: `BaseAnalyzer`, `NLTKAnalyzer`, `GPT2Analyzer`, `RoBERTaAnalyzer`, `EnsembleAnalyzer`. -- Depends on: `src.config.settings`, `src.models.result`, `src.utils.text_processing`, `src.utils.logging_config`; transformers stack in GPT-2/RoBERTa paths. -- Used by: Streamlit apps, tests, any programmatic `analyzer.analyze(text)` caller. - -**Models:** -- Purpose: Serializable result shapes (`AnalysisResult`, `TextMetrics`, `DetectionScore`). -- Location: `src/models/result.py` -- Contains: `@dataclass` types, `to_dict()` / `to_json()` on `AnalysisResult`. -- Depends on: `Verdict`, `ConfidenceLevel` from `src/config/settings.py`. -- Used by: All analyzers and UI layers that render or export results. - -**Configuration:** -- Purpose: Enums, frozen threshold and subsystem configs, cached `Settings` singleton. -- Location: `src/config/settings.py` -- Contains: `Verdict`, `ConfidenceLevel`, `ThresholdConfig`, `NLTKConfig`, `GPT2Config`, `VisualizationConfig`, `Settings`, `get_settings()` (`@lru_cache`). -- Depends on: `os.environ` for `AI_DETECTOR_DEBUG`, `AI_DETECTOR_LOG_LEVEL`. -- Used by: `BaseAnalyzer`, `ChartGenerator`, apps. - -**Infrastructure utilities:** -- Purpose: Logging setup, text cleaning/tokenization/metrics, optional NLTK downloads, plotting. -- Location: `src/utils/logging_config.py`, `src/utils/text_processing.py`, `src/utils/visualization.py` -- Contains: `TextProcessor` (classmethods for corpus-safe NLP), `ChartGenerator` (Plotly primary, Matplotlib `Agg` backend). -- Depends on: NLTK, Plotly, Matplotlib, NumPy; `get_settings()` where needed. -- Used by: Analyzers and Streamlit tabs. +1. **Presentation** — `app.py` (NLTK), `gpt2_app.py` (GPT-2; renamed from + `test.py`), `ensemble.py`. Each prepends `src/` to `sys.path`, configures the + page, injects shared CSS, wires a specific analyzer, and renders an + `AnalysisResult`. Shared UI lives in **`src/ui/`** (styles + components). +2. **Analyzers** — `src/analyzers/`: `BaseAnalyzer`, `NLTKAnalyzer`, + `GPT2Analyzer`, `RoBERTaAnalyzer`, `BinocularsAnalyzer`, `EnsembleAnalyzer`, + plus `calibration.py` (perplexity → AI-probability logistic). +3. **Models/Contract** — `src/models/result.py`: `AnalysisResult`, `TextMetrics`, + `DetectionScore` dataclasses with `to_dict()`/`to_json()`. +4. **Config** — `src/config/settings.py`: frozen dataclasses (`ThresholdConfig`, + `NLTKConfig`, `GPT2Config`, `RoBERTaConfig`, `BinocularsConfig`, + `EnsembleConfig`, `VisualizationConfig`) + `get_settings()` `@lru_cache` + singleton; enums `Verdict`, `ConfidenceLevel`, `DetectionMethod`. +5. **Utils** — `src/utils/`: `text_processing.py` (`TextProcessor`), + `visualization.py` (`ChartGenerator`), `ui_contract.py` (shared UI copy), + `logging_config.py`. +6. **Evaluation** — `src/evaluation/`: `metrics.py` (pure-NumPy accuracy/PR/F1/ + ROC-AUROC/FPR-FNR/ECE), `dataset.py` (JSONL loader over `data/benchmark/`), + `benchmark.py` (runner + `--analyzer {nltk,gpt2,binoculars,ensemble}` CLI + + ROC/calibration plots). ## Data Flow -**Single-analyzer (NLTK / GPT-2) path:** - -1. UI collects `text` and calls `analyzer.analyze(text)` (e.g. `NLTKAnalyzer` from `app.py` via `load_analyzer`). -2. `BaseAnalyzer.analyze()` in `src/analyzers/base_analyzer.py` cleans with `TextProcessor.clean_text()`, sets `text_length`, handles empty/short input warnings. -3. `TextProcessor.compute_metrics()` fills `result.metrics` (`TextMetrics`). -4. Subclass `_perform_analysis()` computes model-specific signals and populates perplexity, burstiness, lexical diversity, sentence variance, and `DetectionScore` entries. -5. `_determine_verdict()` maps weighted feature scores to `Verdict` and `ConfidenceLevel`; `_generate_explanation()` builds the narrative string. -6. UI reads `AnalysisResult` fields and passes word frequencies / scores to `ChartGenerator` for Plotly figures. - -**Ensemble path:** - -1. `EnsembleAnalyzer.analyze()` in `src/analyzers/ensemble_analyzer.py` duplicates early validation/metrics steps then lazy-loads `RoBERTaAnalyzer`, `GPT2Analyzer`, `NLTKAnalyzer` via properties. -2. Component analyzers run (RoBERTa weight may be zeroed by default); scores are fused using configured weights. -3. Returns a single `AnalysisResult` consistent with the shared model type. - -**State Management:** -- No server-side session store; Streamlit reruns the script. Heavy objects (analyzer, `ChartGenerator`) are cached with `@st.cache_resource` in each app file. -- Settings are process-wide via `get_settings()` cache. +`text → TextProcessor.clean_text → _apply_input_cap (max 50k chars) → +compute_metrics → _perform_analysis (per analyzer) → _determine_verdict → +_generate_explanation → AnalysisResult`. The ensemble instead runs GPT-2 + NLTK +(+ optionally RoBERTa/Binoculars when weighted), maps each to a **calibrated** +AI-probability via `logistic_ai_probability` (per-analyzer midpoint = decision +boundary), and fuses them by weight (clamped to `[0,1]`). ## Key Abstractions -**`BaseAnalyzer`:** -- Purpose: Uniform pipeline for text-in → `AnalysisResult` out. -- Examples: `src/analyzers/base_analyzer.py` -- Pattern: Template Method — `analyze()` is concrete; `_perform_analysis()` abstract. - -**`TextProcessor`:** -- Purpose: Single place for NLTK data guarantees, tokenization, and `TextMetrics` computation. -- Examples: `src/utils/text_processing.py` -- Pattern: Stateful class with class-level caches (`_nltk_initialized`, stopwords). - -**`AnalysisResult` / `DetectionScore`:** -- Purpose: Stable contract between analysis and presentation/API docs. -- Examples: `src/models/result.py` -- Pattern: Dataclasses with helper methods (`add_warning`, `add_score`, `to_dict`). - -**`ChartGenerator`:** -- Purpose: Decouple Plotly/Matplotlib construction from Streamlit layout. -- Examples: `src/utils/visualization.py` -- Pattern: Small service class configured from `VisualizationConfig`. +- **BaseAnalyzer** (Template Method) — one abstract hook, shared validation/ + scoring/explanation/timing; input-size cap in `_apply_input_cap`. +- **AnalysisResult** — stable serializable contract between analysis and UI/eval. +- **EnsembleConfig calibration** — decision boundaries and fusion weights live in + config, derived empirically from benchmark perplexity separation (GPT-2 + 0.75 / NLTK 0.25; RoBERTa & Binoculars 0.0, gated). +- **Lazy transformer imports** — `src/analyzers/__init__.py::__getattr__` defers + `GPT2Analyzer`/`RoBERTaAnalyzer`/`BinocularsAnalyzer`/`EnsembleAnalyzer` so + `app.py` runs without `torch`. +- **Shared UI** — `src/ui/components.py` (`render_verdict_card`, + `render_warnings`, `render_footer`, `render_error`, verdict maps; HTML-escaped) + and `src/ui/styles.py` (`BASE_CSS` + `inject_css`). ## Entry Points -**NLTK Streamlit app:** -- Location: `app.py` -- Triggers: `streamlit run app.py` -- Responsibilities: NLTK-only UX; `load_analyzer(ngram_size)` → `NLTKAnalyzer`. - -**Ensemble Streamlit app:** -- Location: `ensemble.py` -- Triggers: `streamlit run ensemble.py` -- Responsibilities: `EnsembleAnalyzer` UX; combined GPT-2 + NLTK (+ optional RoBERTa). - -**GPT-2 Streamlit app:** -- Location: `test.py` -- Triggers: `streamlit run test.py` -- Responsibilities: `GPT2Analyzer`-only UX (despite the name `test.py`). - -**Library / programmatic use:** -- Documented in `docs/API.md`; import analyzers from `src.analyzers` and call `.analyze(text)`. - -**Packaging:** -- Location: `setup.py` — `packages=find_packages(where="src")`, `package_dir={"": "src"}` so the installable name maps to the `src` tree. - -## Error Handling - -**Strategy:** Defensive defaults inside `BaseAnalyzer.analyze()`: exceptions in the try-block set `Verdict.UNCERTAIN`, zero confidence, append warning, generic explanation; logged with `logger.error(..., exc_info=True)`. Streamlit layers wrap display paths in `try/except` and call `st.error`. - -**Patterns:** -- Empty/short text: early return with warnings, no model call. -- NLTK corpus missing: `NLTKAnalyzer._build_model` raises `RuntimeError` with download hint after logging. - -## Cross-Cutting Concerns - -**Logging:** `setup_logging` / `get_logger` from `src/utils/logging_config.py` at app startup and in analyzers. - -**Validation:** Length thresholds from `ThresholdConfig` in `src/config/settings.py`; cleaning in `TextProcessor.clean_text()`. - -**Authentication:** Not applicable — local Streamlit apps, no auth layer in repo. +- `streamlit run app.py` → `NLTKAnalyzer` (lightest, torch-free). +- `streamlit run gpt2_app.py` → `GPT2Analyzer`. +- `streamlit run ensemble.py` → `EnsembleAnalyzer`. +- `python -m src.evaluation.benchmark ...` → benchmark CLI. +- Programmatic: `from src.analyzers. import ; analyzer.analyze(text)`. ---- +## Error Handling Contract -*Architecture analysis: 2026-04-02* +All analysis exceptions are caught inside `analyze()`: set `Verdict.UNCERTAIN`, +zero confidence, log full traceback (`exc_info=True`), and append a **generic** +warning (no exception strings). UIs render errors via `src.ui.render_error` +(generic message to user, full trace to server log). diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md index 15dae3c..e0523ab 100644 --- a/.planning/codebase/CONCERNS.md +++ b/.planning/codebase/CONCERNS.md @@ -1,167 +1,75 @@ -# Codebase Concerns - -**Analysis Date:** 2026-04-02 - -## Tech Debt - -**Triplicated Streamlit entrypoints:** -- Issue: Three large, parallel UIs (`app.py`, `test.py`, `ensemble.py`) repeat CSS, layout, verdict mapping, and chart wiring (~544–585 lines each). Changes require editing multiple files. -- Files: `app.py`, `test.py`, `ensemble.py` -- Impact: Higher defect rate, inconsistent UX, harder refactors. -- Fix approach: Extract shared Streamlit components and styling into a small package under `src/ui/` or a single `pages/` multi-page app; keep one analyzer-specific wiring layer. - -**Import path mutation:** -- Issue: Each root script does `sys.path.insert(0, ...)` before importing `src.*`, instead of an installable package layout. -- Files: `app.py`, `ensemble.py`, `test.py` -- Impact: Fragile runs from wrong CWD, awkward testing and packaging. -- Fix approach: Install in editable mode via `setup.py` / `pyproject.toml` and use normal imports; document `pip install -e .`. - -**Configuration docs vs implementation:** -- Issue: `src/config/settings.py` module docstring references Pydantic; implementation uses `@dataclass` and `os.getenv`. `pydantic` is listed in `requirements.txt` but is not the active settings mechanism. -- Files: `src/config/settings.py`, `requirements.txt` -- Impact: Misleading onboarding; unused or underused dependency. -- Fix approach: Align docstring with dataclasses, or migrate settings to Pydantic v2 and drop redundant prose. - -**RoBERTa included in ensemble while “disabled”:** -- Issue: `EnsembleAnalyzer.weights["roberta"]` is `0.0`, but `analyze()` still runs `self.roberta_analyzer.analyze()` every time, which loads `roberta-base` and runs inference (`src/analyzers/ensemble_analyzer.py`). UI copy in `ensemble.py` says RoBERTa is disabled, yet `load_analyzer()` forces `_ = analyzer.roberta_analyzer`. -- Files: `src/analyzers/ensemble_analyzer.py`, `ensemble.py`, `src/analyzers/roberta_analyzer.py` -- Impact: Large download, RAM, and latency for no contribution to the weighted score; contradicts user expectations. -- Fix approach: Skip RoBERTa entirely when weight is 0 (no load, no forward pass); optionally lazy-load only when weight > 0. - -**Agreement logic uses RoBERTa vote at zero weight:** -- Issue: `_determine_verdict` uses `result.scores[1:]` for `ai_votes` / `agreement`, which includes the RoBERTa row even when its weight is 0. Random/untrained RoBERTa outputs can still move confidence via agreement. -- Files: `src/analyzers/ensemble_analyzer.py` -- Impact: Confidence and narrative (“mixed signals”) can be polluted by a signal explicitly excluded from the blend. -- Fix approach: Only count analyzers with non-zero weight, or omit RoBERTa scores from the list when disabled. - -**Fragile score ordering:** -- Issue: Ensemble logic assumes `result.scores[0]` is the combined “Ensemble AI Score” and `scores[1:]` are individuals (`_determine_verdict`, `_generate_ensemble_explanation`, `ensemble.py` comparison table skips index 0 with `result.scores[1:]`). -- Files: `src/analyzers/ensemble_analyzer.py`, `ensemble.py` -- Impact: Reordering or inserting scores breaks verdict math and UI tables silently. -- Fix approach: Address scores by name or enum, not list index. - -## Known Bugs - -**Example text injection ordering in ensemble empty state:** -- Issue: `ensemble.py` assigns `text_input = st.session_state.pop("text_example")` after the main `elif not text_input` block; Streamlit’s execution model may not repopulate the text area from that assignment in the same run users expect (pattern differs from a `value=` binding). Risk of confusing or ineffective “Use example” flow. -- Files: `ensemble.py` -- Trigger: Click example buttons when the text area is empty. -- Workaround: Paste examples manually. - -**NLTK bootstrap may mark success too early:** -- Issue: `TextProcessor.ensure_nltk_data()` sets `_nltk_initialized = True` after the loop even if some `nltk.download` calls failed (failures are only logged). -- Files: `src/utils/text_processing.py` -- Trigger: Offline or blocked download hosts. -- Workaround: Pre-install corpora; run `nltk.download` manually. - -## Security Considerations - -**Local Streamlit and error surfaces:** -- Risk: `st.error(f"... {str(e)}")` in `app.py`, `ensemble.py`, and `test.py` exposes exception strings to anyone with UI access. -- Files: `app.py`, `ensemble.py`, `test.py` -- Current mitigation: Typical single-user local use. -- Recommendations: Log full trace server-side; show generic messages in UI for production deployments. - -**HTML injection via `unsafe_allow_html`:** -- Risk: Verdict and metrics use fixed templates; warnings rendered as `
⚠️ {warning}
` could inject HTML if a future code path places unsanitized user-controlled text into `result.warnings`. -- Files: `app.py`, `ensemble.py`, `test.py` -- Current mitigation: Warnings are generated internally today. -- Recommendations: Escape user-derived strings before embedding in HTML, or use Streamlit components without raw HTML. - -**Model download trust boundary:** -- Risk: `transformers` loads `gpt2`, `gpt2` tokenizer, and `roberta-base` from the Hugging Face Hub on first run (`src/analyzers/gpt2_analyzer.py`, `src/analyzers/roberta_analyzer.py`). -- Files: `src/analyzers/gpt2_analyzer.py`, `src/analyzers/roberta_analyzer.py` -- Current mitigation: Standard ecosystem defaults; cache dir configurable via `GPT2Config.cache_dir` in settings. -- Recommendations: Pin revisions, verify checksums, or vendor weights for high-assurance environments. - -**Deployment and DoS:** -- Risk: No authentication, rate limits, or input size caps at the app layer; very long pasted text increases GPU/CPU time for GPT-2 sliding windows (`max_token_length` 1024, stride 512 in `src/config/settings.py`). -- Files: `src/analyzers/gpt2_analyzer.py`, Streamlit entrypoints -- Current mitigation: Documented as local processing. -- Recommendations: Add max character limits and server-side timeouts if exposed beyond localhost. - -## Performance Bottlenecks - -**Ensemble cold start and RoBERTa:** -- Problem: First load pulls and initializes RoBERTa, GPT-2, and NLTK-backed models despite RoBERTa having zero ensemble weight. -- Files: `ensemble.py`, `src/analyzers/ensemble_analyzer.py` -- Cause: Eager `roberta_analyzer` property access in `load_analyzer()` and unconditional `roberta_analyzer.analyze()` in `analyze()`. -- Improvement path: Gate RoBERTa on weight > 0; defer downloads until needed. - -**GPT-2 perplexity sliding window:** -- Problem: `_compute_perplexity_gpt2` iterates with `stride` over full token length—multiple forward passes per request. -- Files: `src/analyzers/gpt2_analyzer.py` -- Cause: Design for long texts vs 1024-token windows. -- Improvement path: Cap analyzed prefix for interactive UI, batch tuning, or GPU-only paths. - -**NLTK model rebuild on n-gram change:** -- Problem: `NLTKAnalyzer.model` rebuilds when `ngram_size` changes (`src/analyzers/nltk_analyzer.py`), which is expensive on the Brown corpus. -- Files: `src/analyzers/nltk_analyzer.py`, `app.py` (sidebar selectbox) -- Cause: Cache key includes n-gram in Streamlit but underlying build still runs when cache misses. -- Improvement path: Precompute or persist models per n. +# Concerns -## Fragile Areas +**Analysis Date:** 2026-07-06 + +> Context: a large audit-remediation (PR #2) was merged, so most previously- +> documented concerns are **resolved** (MLE→smoothed NLTK, ensemble calibration/ +> AI-bias, RoBERTa gating, CI, safetensors, dead deps, UI dedup, input cap, +> error-leakage, `test.py`→`gpt2_app.py`). The items below are what **remains**, +> verified against `main`. -**RoBERTa “AI detection” head:** -- Files: `src/analyzers/roberta_analyzer.py` -- Why fragile: Uses randomly initialized classification head on `roberta-base` (explicit `logger.warning`); logits are not meaningful for AI vs human until fine-tuned. -- Safe modification: Do not increase `weights["roberta"]` until a trained checkpoint is wired via `from_pretrained` to a real detector model. -- Test coverage: `tests/test_roberta_analyzer.py` exists; treat as unit-level only—behavior is not semantically validated against ground truth. +## Validation / ML Credibility (highest priority) -**Ensemble combination constants:** -- Files: `src/analyzers/ensemble_analyzer.py` -- Why fragile: Hard-coded normalization `1 - (perplexity / 500)` for both GPT-2 and NLTK maps different perplexity scales into `[0,1]` the same way. -- Safe modification: Calibrate per analyzer using held-out data; add tests that lock expected ranges. +- **No large-benchmark evaluation.** The bundled set is **24 in-distribution + samples** (`data/benchmark/samples.jsonl`); its perfect scores (AUROC/FPR 0.000) + are calibration/regression signals, **not** an accuracy claim. No RAID/HC3 run + exists. The `benchmark.py --dataset` harness is ready; this is the top gap. +- **RoBERTa slot untrained/disabled.** `roberta-base` has a random classification + head; kept at weight 0. Needs a fine-tuned checkpoint before enabling. +- **Brown-corpus NLTK signal is weak** (1961 corpus; measured AUROC ~0.41 + standalone). Intentionally carries only 0.25 ensemble weight; GPT-2 dominates. -**Large presentation modules:** -- Files: `app.py` (544 non-empty lines), `test.py` (585), `ensemble.py` (567), `src/utils/visualization.py` (366), `src/utils/text_processing.py` (346), `src/analyzers/gpt2_analyzer.py` (341) -- Why fragile: High line count concentrates UI, CSS, and orchestration without tests. -- Safe modification: Split files first; add regression tests on analyzers and `AnalysisResult` serialization. +## Fragile Areas -## Scaling Limits +- **Positional `scores[0]` access** in `src/analyzers/ensemble_analyzer.py` + (2 uses) and `src/analyzers/binoculars_analyzer.py` (1 use). The "index 0 = + primary score" contract is documented but not enforced — reordering score + additions would silently break verdict math. Fix: address scores by name. +- **Ensemble weights must sum to 1** by convention; enabling Binoculars/RoBERTa + requires manual rebalancing. Mitigated by a `[0,1]` clamp on the fused score, + but no auto-normalization. -**Memory and concurrent users:** -- Current capacity: Documented ~2–3 GB (sidebar in `ensemble.py`) to ~4–6 GB on error path; multiple Torch models in one process. -- Limit: OOM or severe slowdown on CPU with ensemble enabled. -- Scaling path: Separate analyzer services, model choice per tier, or drop RoBERTa entirely until needed. +## Security / Supply Chain -## Dependencies at Risk +- **Model Hub revisions default to `None`** (`GPT2Config`, `RoBERTaConfig`, + `BinocularsConfig`). `use_safetensors=True` mitigates pickle RCE, but weights + are not pinned to a commit unless a `revision` is set — reproducibility/supply- + chain gap for high-assurance use. +- **No auth / rate limiting** on the Streamlit apps (documented as local-use). + Input is capped (50k chars) and errors are generic, so the main residual risk + is running an app on an untrusted network without a reverse proxy. -**NumPy major version pin:** -- Risk: `requirements.txt` caps `numpy<2.0.0` while the ecosystem moves toward NumPy 2.x. -- Impact: Future conflicts with newer `torch` / `pandas` wheels. -- Migration plan: Test against NumPy 2.x and relax the upper bound when compatible. +## Dependencies / Build -**Pydantic vs dataclass settings:** -- Risk: Two configuration stories (listed Pydantic, implemented dataclasses) complicate dependency justification. -- Impact: Audit noise and possible version drift. -- Migration plan: Pick one approach and document in `docs/API.md` and README. +- **`numpy<2.0` pin** (`requirements.txt`) is aging vs the ecosystem. +- **Dockerfile base `python:3.9-slim`** is old relative to the new floors + (`torch>=2.6`, `transformers>=4.48`); bump to 3.11 recommended. -## Missing Critical Features +## CI / Tooling Gaps -**Documented public API vs code:** -- Problem: `docs/API.md` only demonstrates `NLTKAnalyzer`; no sections for `GPT2Analyzer`, `EnsembleAnalyzer`, `RoBERTaAnalyzer`, or Streamlit entrypoints. -- Files: `docs/API.md` -- Blocks: Integrators cannot rely on docs for ensemble or GPT-2 usage without reading source. +- **mypy** documented but **not run in CI**. +- **No coverage gate** (`--cov-fail-under` unset) — 92% can erode silently. +- **Slow model tests never run in CI** (no scheduled full-suite job), so + transformer compute paths are only verified locally. -**CI pipeline:** -- Problem: No `.github/workflows` (or equivalent) detected in repo root for automated test runs. -- Blocks: Regressions on `tests/` may go unnoticed. +## Packaging -## Test Coverage Gaps +- Entry scripts use `sys.path.insert(0, ".../src")`; **no `pyproject.toml`** and + no editable install. Documented as an invariant (`CLAUDE.md`), but blocks clean + packaging/distribution. -**Streamlit and end-to-end flows:** -- What’s not tested: User clicks, `st.cache_resource` behavior, and chart rendering. -- Files: `app.py`, `ensemble.py`, `test.py` -- Risk: UI-only regressions ship without detection. -- Priority: Medium +## Documentation Drift -**Cross-analyzer calibration:** -- What’s not tested: Whether ensemble weights and perplexity normalization produce stable verdicts on a fixed golden corpus. -- Files: `src/analyzers/ensemble_analyzer.py` -- Risk: Silent accuracy drift when thresholds change. -- Priority: Medium +- **`CLAUDE.md` (repo root) is stale** and has an uncommitted local edit: still + references `test.py`, 65/35 ensemble weights, and "Pydantic" settings. It is + read by agents as source-of-truth — regenerate (this map refresh is a step). +- `.planning/codebase/` (this folder) was just refreshed to match post-merge + reality. ---- +## Missing Portfolio/Product Features (from audit roadmap) -*Concerns audit: 2026-04-02* +- No Hugging Face Spaces demo, no screenshots/GIF in README, no `examples/`, + no model card, no CODEOWNERS, no `.pre-commit-config.yaml` (pre-commit is in + dev deps but unconfigured), empty repo description, no `v2.0.0` release/tag. +- Longer-term: REST API, batch/PDF-DOCX input, export formats, paraphrase/ + humanizer-attack robustness testing, per-language calibration. diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md index ce00521..9e45fb6 100644 --- a/.planning/codebase/CONVENTIONS.md +++ b/.planning/codebase/CONVENTIONS.md @@ -1,111 +1,66 @@ -# Coding Conventions - -**Analysis Date:** 2026-04-02 - -## Naming Patterns - -**Files:** -- Python modules: `snake_case.py` (e.g. `base_analyzer.py`, `text_processing.py`, `logging_config.py`). -- Streamlit entry scripts at repo root: `app.py`, `test.py`, `ensemble.py` (single-word or short names). -- Tests: `test_.py` under `tests/` (e.g. `tests/test_nltk_analyzer.py`). - -**Functions:** -- `snake_case` for functions and methods (e.g. `get_settings`, `clean_text`, `compute_metrics`). - -**Variables:** -- `snake_case` for locals and instance attributes; short aliases in hot paths are acceptable (e.g. `t = self.thresholds` in `src/analyzers/base_analyzer.py`). - -**Classes:** -- `PascalCase` for classes (e.g. `BaseAnalyzer`, `TextProcessor`, `AnalysisResult`, `NLTKAnalyzer`). - -**Types:** -- Prefer `from __future__ import annotations` and typing imports (`Optional`, `List`, `Dict`, `Tuple`) as in `src/models/result.py` and `src/analyzers/base_analyzer.py`. -- Enums for fixed vocabularies: `Verdict`, `ConfidenceLevel`, `DetectionMethod` in `src/config/settings.py`. - -**Private / internal:** -- Leading underscore for non-public implementation details (e.g. `_perform_analysis`, `_nltk_initialized`, `_roberta_analyzer` on ensemble). - -## Code Style - -**Formatting:** -- **Black** with line length **100** (see `Makefile` target `format`). -- **isort** with **`--profile=black`** alongside Black. - -**Linting:** -- **flake8** on `src/`, `tests/`, `app.py`, `test.py`, `ensemble.py` with `--max-line-length=100` (`Makefile` target `lint`). -- **mypy** on `src/` with `--ignore-missing-imports`. -- **pylint** on `src/` with `--disable=C0114,C0115,C0116` (docstring-related disables). - -**Declared dev stack:** `requirements-dev.txt` lists black, flake8, isort, mypy, pylint, pre-commit (no committed `pyproject.toml` or `.pre-commit-config.yaml` detected in repo root). - -## Import Organization - -**Order (observed):** -1. Standard library (`from __future__`, `abc`, `typing`, `os`, etc.). -2. Third-party (`nltk`, `streamlit`, `pytest` in tests). -3. First-party: `from src....` (package layout under `src/`). - -**Path resolution for `src`:** -- **Makefile** sets `PYTHONPATH=src` for `run-*`, `test`, and `lint` targets. -- **Streamlit apps** (`app.py`, and same pattern in `test.py` / `ensemble.py`): `sys.path.insert(0, ... "src")` then `from src....`. -- **Tests:** `tests/conftest.py` inserts `../src` on `sys.path` so `from src....` resolves when pytest is run without `PYTHONPATH`. - -**Prescriptive rule:** Prefer running tests and apps the way `Makefile` documents (`PYTHONPATH=src` or path insert) so `import src` remains consistent. +# Conventions + +**Analysis Date:** 2026-07-06 + +## Code Style & Tooling + +- **Black**, line length **100** (`Makefile`, CI). +- **isort** with `--profile=black`. +- **flake8** `--max-line-length=100` on `src/`, `tests/`, `app.py`, + `gpt2_app.py`, `ensemble.py`. Config in `.flake8`: `extend-ignore = E203, W503` + (Black-compatible) and `per-file-ignores` granting `E402` to the three entry + scripts (they must `sys.path.insert` before importing `src.*`). +- **mypy** documented (`make lint` / README) with `--ignore-missing-imports`, but + **not currently enforced in CI**. +- The **whole tree passes** black + isort + flake8 (enforced by CI on every push/PR). + +## Naming + +- `snake_case` modules/functions/vars; `PascalCase` classes; leading underscore + for non-public methods (`_perform_analysis`, `_apply_input_cap`, + `_combine_results`). +- Enums for fixed vocabularies (`Verdict`, `ConfidenceLevel`, `DetectionMethod`). + +## Patterns + +- `from __future__ import annotations` + `typing` imports across modules. +- **Config:** frozen `@dataclass(frozen=True)` blobs; mutable `Settings` behind + `get_settings()` `@lru_cache` singleton. **Never** instantiate `Settings()` + directly; **never** hardcode threshold values — use `self.thresholds.*`. +- **Analyzer contract:** only implement `_perform_analysis`; `analyze()` is + inherited (except `EnsembleAnalyzer`, which overrides it). Populate + `result.perplexity/burstiness/lexical_diversity/sentence_variance` and add + `DetectionScore`s. +- **Lazy imports:** torch-backed analyzers are loaded via + `src/analyzers/__init__.py::__getattr__` — keep them out of the eager block so + `app.py` stays torch-free. +- **Intent layer:** `AGENTS.md` files under `src/analyzers/`, `src/models/`, + `src/utils/` document local invariants — read before editing. ## Error Handling -**Analyzer pipeline (`src/analyzers/base_analyzer.py`):** -- Broad `try`/`except Exception` around analysis: on failure, log with `logger.error(..., exc_info=True)`, set `Verdict.UNCERTAIN`, zero confidence, append warning, set a safe `explanation`. -- Empty or invalid text: early return with structured `AnalysisResult` (no exception). - -**Validation-style errors:** -- `NLTKAnalyzer.set_ngram_size` raises `ValueError` for invalid sizes; tests expect `pytest.raises(ValueError)` (`tests/test_nltk_analyzer.py`). - -**NLTK downloads (`src/utils/text_processing.py`):** -- Per-package download failures: log warning, continue where possible. - -**Prescriptive pattern for new analyzers:** Subclass `BaseAnalyzer`, implement `_perform_analysis`; do not bypass `analyze()` validation and error envelope unless there is a strong reason. - -## Logging - -**Framework:** Standard library `logging`, wrapped by `src/utils/logging_config.py`. - -**Patterns:** -- `get_logger(__name__)` at module level; use `logger.info` / `logger.error` / `logger.warning` as in `base_analyzer.py` and `text_processing.py`. -- `setup_logging(level, log_file=None)` configures format `%(asctime) | %(levelname) | %(name)s:%(funcName)s:%(lineno)d | %(message)s` and quiets noisy third-party loggers (`transformers`, `torch`, `urllib3`, `filelock`). - -**Settings:** `Settings.log_level` and `debug` can be driven by env vars `AI_DETECTOR_LOG_LEVEL` and `AI_DETECTOR_DEBUG` (`src/config/settings.py`). - -## Comments - -**Module docstrings:** -- Top-of-file banner style with title lines (`===`) in several modules (e.g. `src/utils/text_processing.py`, `src/config/settings.py`). - -**Class/method docstrings:** -- Google-style **Args** / **Returns** blocks on public methods (e.g. `BaseAnalyzer.analyze`, `setup_logging`). - -**Inline:** -- Section dividers in Streamlit UI code (e.g. `# ─── Setup ───` in `app.py`). - -## Function Design - -**Size:** `BaseAnalyzer` centralizes orchestration; subclasses focus on `_perform_analysis`. Prefer keeping verdict logic in the base class unless the method differs fundamentally. - -**Parameters:** Explicit typed parameters; dataclass/`field(default_factory=...)` for mutable defaults (`src/models/result.py`). - -**Return values:** `AnalysisResult` (or domain dataclasses) rather than loose dicts for core API; `to_dict()` / `to_json()` for serialization. +- Broad `try/except Exception` around analysis; on failure log with + `exc_info=True`, set `Verdict.UNCERTAIN`, zero confidence, and append a + **generic** warning (exception strings are never surfaced to the UI). +- Empty/short input: early structured return with warnings, no model call. +- UIs use `src.ui.render_error(exc)` — logs full trace, shows a generic escaped + message. `NLTKAnalyzer.set_ngram_size` raises `ValueError` for invalid sizes. -## Module Design +## Security-conscious conventions -**Exports:** -- Package `__init__.py` files under `src/` exist for package structure; import concrete symbols from submodules (e.g. `from src.analyzers.nltk_analyzer import NLTKAnalyzer`). +- Model weights load with `use_safetensors=True`; Hub `revision` is pinnable via + config. All user-derived strings are `html.escape`d before `unsafe_allow_html`. + Input is capped at `ThresholdConfig.max_input_chars` (50000). -**Configuration:** -- Immutable threshold/config blobs: `@dataclass(frozen=True)` (`ThresholdConfig`, `NLTKConfig`, etc.) in `src/config/settings.py`. -- Mutable `Settings` uses `@lru_cache` singleton via `get_settings()`. +## Comments & Docstrings -**Barrel files:** Not heavily used; prefer explicit imports from feature modules. +- Google-style Args/Returns on public methods; top-of-file banner docstrings; + section dividers (`# ─── ... ───`) in Streamlit code. Comments explain + constraints/rationale, not narration. ---- +## Commits -*Convention analysis: 2026-04-02* +- Small, focused, Conventional-Commit style (`fix(ensemble): ...`, + `refactor(ui): ...`), often referencing an audit finding (C1/H3/M4). PRs + include benchmark numbers when detection behaviour changes + (`.github/PULL_REQUEST_TEMPLATE.md`). diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md index 40e7a04..b97f6c1 100644 --- a/.planning/codebase/INTEGRATIONS.md +++ b/.planning/codebase/INTEGRATIONS.md @@ -1,77 +1,53 @@ -# External Integrations - -**Analysis Date:** 2026-04-02 - -## APIs & External Services - -**Model & artifact downloads (Hugging Face Hub):** -- GPT-2 — `GPT2TokenizerFast.from_pretrained` and `GPT2LMHeadModel.from_pretrained` in `src/analyzers/gpt2_analyzer.py` using `settings.gpt2.model_name` (default `gpt2` from `src/config/settings.py`). Optional `cache_dir` from `GPT2Config`. -- RoBERTa — `RobertaTokenizer.from_pretrained("roberta-base")` and sequence classification model load in `src/analyzers/roberta_analyzer.py`. Ensemble notes RoBERTa disabled by weight `0.0` in `src/analyzers/ensemble_analyzer.py` until fine-tuning. -- Network: First run (or empty cache) pulls weights from the public Hugging Face Hub; `transformers`/`huggingface_hub` handle HTTP. Docker Compose mounts `model_cache` to `/home/appuser/.cache` for `gpt2-detector` and `ensemble-detector` in `docker-compose.yml`. - -**NLTK data:** -- Remote download via `nltk.download(...)` — Dockerfile runs downloads for punkt, punkt_tab, stopwords, brown, averaged_perceptron_tagger. `NLTKConfig.required_data` in `src/config/settings.py` matches expected packages. Compose service `nltk-detector` persists `nltk_data` volume at `/home/appuser/nltk_data`. - -**No first-party REST API server:** -- The product is Streamlit apps, not a separate HTTP JSON API. Programmatic use is in-process Python (see `docs/API.md` for analyzer imports). Do not assume OpenAPI routes unless added later. - -## Data Storage - -**Databases:** -- Not applicable — No SQL/NoSQL clients, ORMs, or connection strings in application code. - -**File Storage:** -- Local filesystem and container volumes only — Application code, NLTK data dir, Hugging Face cache under user home (`.cache`) in containers per `docker-compose.yml`. - -**Caching:** -- Hugging Face/transformers model cache on disk (path influenced by `cache_dir` in settings and default cache layout). -- Streamlit `@st.cache_resource` in `app.py` (and analogous patterns in other entry files) caches analyzer and chart generator instances in process memory. - -## Authentication & Identity - -**Auth Provider:** -- Not applicable for core app — Streamlit apps are unauthenticated single-user sessions unless deployed behind a reverse proxy or platform IAM. No OAuth, API keys, or session stores in `src/`. - -## Monitoring & Observability - -**Error Tracking:** -- None integrated — No Sentry, Rollbar, or similar in `requirements.txt` or imports. - -**Logs:** -- Standard library `logging` configured in `src/utils/logging_config.py` (stdout; optional file path). Levels for `transformers`, `torch`, `urllib3`, `filelock` reduced to WARNING to limit noise. - -## CI/CD & Deployment - -**Hosting:** -- Documented options in `docs/DEPLOYMENT.md` (Docker, Heroku, AWS, Azure, GCP); not wired as code in this repo snapshot. - -**CI Pipeline:** -- Not detected — No `.github/workflows/` or similar in workspace. - -## Environment Configuration - -**Required env vars:** -- None strictly required for local run beyond Python/Streamlit defaults. Optional: `AI_DETECTOR_DEBUG`, `AI_DETECTOR_LOG_LEVEL` in `src/config/settings.py`. Docker sets `PYTHONPATH`, Streamlit server env vars in `Dockerfile` / `docker-compose.yml`. - -**Secrets location:** -- No application secrets required for model download of public GPT-2/roberta-base weights. If adding private Hub tokens or paid APIs, use platform secret stores or `.env` (not committed); `docs/DEPLOYMENT.md` mentions a `.env` example with `LOG_LEVEL`, `NGRAM_ORDER`, etc., which may not match `get_settings()` — treat docs as aspirational until aligned with `src/config/settings.py`. - -## Webhooks & Callbacks - -**Incoming:** -- None — No webhook endpoints; Streamlit exposes UI and internal health URL only. - -**Outgoing:** -- None for business logic — Only implicit outbound HTTP from `transformers`/NLTK when fetching models or data. - -## Third-Party Python stack notes - -**HTTP stack:** -- `urllib3` appears only as a logging noise reducer in `src/utils/logging_config.py` (dependency of libraries that perform downloads). - -**Streamlit health:** -- `Dockerfile` `HEALTHCHECK` uses `curl` against `http://localhost:8501/_stcore/health`. - ---- - -*Integration audit: 2026-04-02* +# Integrations + +**Analysis Date:** 2026-07-06 + +The project is **local-first**: core detection requires no third-party API calls +and no user text leaves the machine. The only external dependency at runtime is +the Hugging Face Hub, used to download model weights on first use. + +## External Model Downloads (Hugging Face Hub) + +- **GPT-2** (`gpt2`) — downloaded by `src/analyzers/gpt2_analyzer.py` via + `GPT2LMHeadModel.from_pretrained(..., use_safetensors=True, revision=...)`. + ~500 MB, cached after first run. +- **DistilGPT-2** (`distilgpt2`) — the Binoculars "performer" model in + `src/analyzers/binoculars_analyzer.py`; observer is `gpt2`. Both share the + GPT-2 tokenizer. +- **RoBERTa** (`roberta-base`) — `src/analyzers/roberta_analyzer.py`. Loaded + only when `EnsembleConfig.weight_roberta > 0` (disabled by default: an + untrained classification head). Uses `use_safetensors=True`. +- Loading is hardened: `use_safetensors=True` avoids pickle deserialization, and + a configurable `revision` (in `GPT2Config` / `RoBERTaConfig` / + `BinocularsConfig`) allows pinning a Hub commit. **Revisions default to + `None`** (tracks the default branch) — pin a commit hash for high-assurance + deployments. + +## NLTK Corpora (downloaded, then local) + +`src/utils/text_processing.py::TextProcessor.ensure_nltk_data()` downloads +`punkt`, `punkt_tab`, `stopwords`, `brown`, `averaged_perceptron_tagger` on first +use (retried on failure; only marks initialized when all are present). The +`Dockerfile` pre-downloads these at build time. + +## Databases / Auth / Webhooks + +- **None.** No database, no authentication provider, no message queue, no + webhooks, no outbound telemetry. Streamlit's usage stats are disabled + (`STREAMLIT_BROWSER_GATHER_USAGE_STATS=false` in Docker; `.streamlit/config.toml`). + +## CI / Dev-time Integrations + +- **GitHub Actions** (`.github/workflows/ci.yml`) — lint (flake8/black/isort) + + pytest matrix (Python 3.9/3.10/3.11), `pip` cache, NLTK data download step. +- **GitHub** repo `satyamshivam13/AI_Text_Detector` — PR-based workflow; automated + reviewers (Copilot, GitGuardian) run on PRs. + +## Deployment Surfaces + +- **Docker** (`Dockerfile`, `docker-compose.yml`) — single image, three Compose + services (`nltk-detector`, `gpt2-detector`, `ensemble-detector`) differing only + by the Streamlit entry script and resource limits. Health check curls + `http://localhost:8501/_stcore/health`. +- **Procfile** — present for PaaS (e.g. Heroku-style) `streamlit run` deploys. +- No cloud provider SDKs, secrets managers, or IaC are wired in. diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md index b7a88f8..cf1fd1f 100644 --- a/.planning/codebase/STACK.md +++ b/.planning/codebase/STACK.md @@ -1,74 +1,93 @@ # Technology Stack -**Analysis Date:** 2026-04-02 +**Analysis Date:** 2026-07-06 ## Languages **Primary:** -- Python 3 — Application, analyzers, Streamlit UIs; `setup.py` declares `python_requires=">=3.8"` with classifiers through 3.11; `Dockerfile` pins runtime image `python:3.9-slim`. +- Python 3 - Entire codebase: analyzers (`src/analyzers/`), evaluation harness (`src/evaluation/`), Streamlit UIs (`app.py`, `gpt2_app.py`, `ensemble.py`), shared UI helpers (`src/ui/`), utilities (`src/utils/`) **Secondary:** -- Not applicable — No TypeScript, JavaScript bundles, or separate frontend beyond Streamlit-rendered HTML/CSS in `app.py`, `test.py`, and `ensemble.py`. +- Not applicable - No JavaScript/TypeScript; all HTML/CSS is inline strings rendered through Streamlit (`src/ui/styles.py`, `src/ui/components.py`) + +**Version note:** `setup.py` declares `python_requires=">=3.8"` with classifiers through 3.11, but CI (`.github/workflows/ci.yml`) tests only 3.9/3.10/3.11 and `src/utils/logging_config.py` uses `list[logging.Handler]` runtime annotations (3.9+ syntax). Treat 3.9 as the effective floor. ## Runtime **Environment:** -- CPython (see Docker: `python:3.9-slim` in `Dockerfile`). +- CPython 3.9+ (Docker image pins `python:3.9-slim` in `Dockerfile`; CI matrix runs 3.9, 3.10, 3.11) +- GPU optional: `GPT2Analyzer.device` auto-selects CUDA when available, else CPU (`src/analyzers/gpt2_analyzer.py`) **Package Manager:** -- pip — Used in `Dockerfile` (`pip install -r requirements.txt`). -- Lockfile: Not detected — No `poetry.lock`, `Pipfile.lock`, or `uv.lock` in repo; pin ranges live in `requirements.txt`. +- pip (used in `Dockerfile`, `Makefile`, CI) +- Lockfile: Not detected - version ranges only, pinned in `requirements.txt` (no `poetry.lock`/`Pipfile.lock`/`uv.lock`) ## Frameworks **Core:** -- Streamlit (>=1.28,<2) — Web UI; entry via `streamlit run` with `app.py` (NLTK), `test.py` (GPT-2), or `ensemble.py` (ensemble). See `docker-compose.yml` service `command` overrides. -- PyTorch (`torch` >=2,<3) — Device selection and model inference in `src/analyzers/gpt2_analyzer.py` and `src/analyzers/roberta_analyzer.py`. -- Hugging Face Transformers (`transformers` >=4.35,<5) — `GPT2LMHeadModel`, `GPT2TokenizerFast`, `RobertaTokenizer`, `RobertaForSequenceClassification` in those analyzers. -- NLTK (>=3.8,<4) — N-gram / Brown corpus analysis in `src/analyzers/nltk_analyzer.py`. +- Streamlit >=1.28,<2 - Sole web/UI framework; three entry points run via `streamlit run`: `app.py` (NLTK), `gpt2_app.py` (GPT-2), `ensemble.py` (ensemble) +- PyTorch (`torch`) >=2.6,<3 - Model inference and device selection in `src/analyzers/gpt2_analyzer.py`, `src/analyzers/roberta_analyzer.py`, `src/analyzers/binoculars_analyzer.py` +- Hugging Face Transformers >=4.48,<5 - `GPT2LMHeadModel`, `GPT2TokenizerFast`, `RobertaTokenizer`, `RobertaForSequenceClassification`; all `from_pretrained` calls pass `use_safetensors=True` and a pinnable `revision` (supply-chain hardening; floors chosen to clear known deserialization advisories per comments in `requirements.txt`) +- NLTK >=3.8.1,<4 - Brown-corpus n-gram language model in `src/analyzers/nltk_analyzer.py`; tokenization/metrics in `src/utils/text_processing.py` **Testing:** -- pytest (>=7.4) — Declared in `requirements-dev.txt` and `setup.py` `extras_require.dev`; tests under `tests/`. +- pytest >=7.4 with pytest-cov, pytest-mock, pytest-asyncio (`requirements-dev.txt`) +- `pytest.ini` defines a `slow` marker; CI runs `-m "not slow"` with `--cov=src` +- Test suite: `tests/` (23 test modules covering analyzers, evaluation, UI contracts, Streamlit apps) **Build/Dev:** -- setuptools — Package layout in `setup.py` with `packages=find_packages(where="src")`, `package_dir={"": "src"}`. -- black, flake8, isort, mypy, pylint, pre-commit — Listed in `requirements-dev.txt`. -- Sphinx + sphinx-rtd-theme — Documentation tooling in `requirements-dev.txt`. +- setuptools via `setup.py` - `packages=find_packages(where="src")`, `package_dir={"": "src"}`, version 2.0.0 +- black (line length 100), isort (`--profile=black`), flake8 (`.flake8`: max 100, ignores E203/W503, per-file E402 exemptions for the three Streamlit entry points), mypy (`--ignore-missing-imports`), pylint - orchestrated by `Makefile` targets `format`/`lint` +- pre-commit >=3.3.0 declared in `requirements-dev.txt` (no `.pre-commit-config.yaml` detected at repo root) +- Sphinx + sphinx-rtd-theme declared in `requirements-dev.txt` (docs tooling; no `docs/conf.py` build detected) ## Key Dependencies **Critical:** -- `streamlit` — Sole HTTP-serving application layer; configuration in `.streamlit/config.toml`. -- `torch` + `transformers` — GPT-2 and RoBERTa model loading (`from_pretrained`) in `src/analyzers/gpt2_analyzer.py`, `src/analyzers/roberta_analyzer.py`. -- `nltk` — Corpus and tokenizer resources; Dockerfile pre-downloads punkt, punkt_tab, stopwords, brown, averaged_perceptron_tagger. +- `streamlit` - The only serving layer; configured via `.streamlit/config.toml` +- `torch` + `transformers` - GPT-2 (`gpt2`), DistilGPT-2 (`distilgpt2`, Binoculars performer), RoBERTa (`roberta-base`) loading; safetensors-only weight loading enforced in all three transformer analyzers +- `nltk` - Brown corpus n-gram model; required data packages listed in `NLTKConfig.required_data` (`src/config/settings.py`): punkt, punkt_tab, stopwords, brown, averaged_perceptron_tagger -**Data & visualization:** -- `numpy`, `pandas` — Numerical/tabular use in analyzers and utilities. -- `matplotlib` (Agg backend in `src/utils/visualization.py`), `plotly` — Charts for Streamlit (`st.plotly_chart` in `app.py`). +**Infrastructure:** +- `numpy` >=1.24,<2 / `pandas` >=2,<3 - Numeric/tabular computation in analyzers, evaluation metrics (`src/evaluation/metrics.py`), and charts +- `plotly` >=5.18,<6 - Primary interactive charts (`src/utils/visualization.py`, rendered via `st.plotly_chart`) +- `matplotlib` >=3.7,<4 - Secondary charting with Agg backend (`src/utils/visualization.py`); calibration/ROC images in `docs/benchmarks/` -**Declared but lightly or unused in source:** -- `pydantic` — Listed in `requirements.txt`; `src/config/settings.py` and `src/models/result.py` use `dataclasses` instead. Prefer aligning future config/models with one approach. -- `python-dotenv`, `structlog` — Present in `requirements.txt`; no `load_dotenv` or `structlog` usage detected in application Python files; logging uses stdlib in `src/utils/logging_config.py`. +**Removed (do not reintroduce without cause):** +- `pydantic`, `structlog`, `python-dotenv` are no longer in `requirements.txt`; config/models use stdlib `dataclasses` (`src/config/settings.py`, `src/models/result.py`) and logging uses stdlib `logging` (`src/utils/logging_config.py`) ## Configuration **Environment:** -- Application toggles: `AI_DETECTOR_DEBUG`, `AI_DETECTOR_LOG_LEVEL` — Read in `Settings.__post_init__` in `src/config/settings.py`. -- Docker / Streamlit: `PYTHONPATH=/app/src`, `STREAMLIT_SERVER_PORT`, `STREAMLIT_SERVER_ADDRESS`, `STREAMLIT_BROWSER_GATHER_USAGE_STATS` — Set in `Dockerfile` and `docker-compose.yml`. -- Streamlit server/theme/browser/logger — `.streamlit/config.toml` (e.g. `headless`, `port`, `maxUploadSize`, XSRF). +- `AI_DETECTOR_DEBUG` (bool, default false) and `AI_DETECTOR_LOG_LEVEL` (default INFO) - read in `Settings.__post_init__` (`src/config/settings.py`) +- All other configuration is code-level frozen dataclasses: `ThresholdConfig`, `NLTKConfig`, `GPT2Config`, `RoBERTaConfig`, `BinocularsConfig`, `EnsembleConfig`, `VisualizationConfig` in `src/config/settings.py`, accessed via the `@lru_cache` singleton `get_settings()` +- Docker/Streamlit env: `PYTHONPATH=/app/src`, `STREAMLIT_SERVER_PORT`, `STREAMLIT_SERVER_ADDRESS`, `STREAMLIT_BROWSER_GATHER_USAGE_STATS=false` (`Dockerfile`, `docker-compose.yml`) +- No `.env` files present; no dotenv loading anywhere **Build:** -- `Dockerfile` — Multi-stage base, system `build-essential` + `curl`, NLTK download step, `ENTRYPOINT ["streamlit", "run"]`, default `CMD ["app.py"]`. -- `docker-compose.yml` — Three services sharing the same image build; different ports and commands for NLTK vs GPT-2 vs ensemble; named volumes `nltk_data` and `model_cache`. +- `setup.py` - package metadata and install +- `requirements.txt` / `requirements-dev.txt` - dependency pins +- `.streamlit/config.toml` - server (headless, port 8501, `maxUploadSize=10`, XSRF on, CORS off), dark theme, telemetry off, logger format +- `Dockerfile` - `python:3.9-slim`, non-root `appuser`, NLTK data pre-download, `ENTRYPOINT ["streamlit", "run"]`, default `CMD ["app.py"]`, health check on `/_stcore/health` +- `docker-compose.yml` - three services (nltk-detector :8501, gpt2-detector :8502, ensemble-detector :8503) sharing one image; named volumes `nltk_data` and `model_cache`; per-service memory/CPU limits (2G/4G/6G) +- `Makefile` - install/run/test/lint/format/docker targets; sets `PYTHONPATH=src` for run/test/lint +- `Procfile` - Heroku-style: `web: streamlit run app.py --server.port=$PORT ...` +- `pytest.ini` - `slow` marker registration ## Platform Requirements **Development:** -- Python >=3.8, pip, virtualenv per `docs/DEPLOYMENT.md`; run `streamlit run app.py` from repo root (see `app.py` docstring). Install from `requirements.txt`; dev extras via `requirements-dev.txt` or `pip install -e ".[dev]"` per `setup.py`. +- Python 3.9+ (effective floor; see version note), pip, virtualenv per `docs/DEPLOYMENT.md` +- NLTK data download required post-install (`make install` runs it; CI and Dockerfile do the same) +- First GPT-2/RoBERTa/Binoculars run downloads model weights from Hugging Face Hub (network needed once, then cached) +- Quality gates before commit: `make format && make lint` (Black 100, isort black-profile, flake8 100) **Production:** -- Container: Linux image exposing port 8501; health check curls `http://localhost:8501/_stcore/health` in `Dockerfile`. GPT-2/ensemble paths need more RAM/CPU per `docker-compose.yml` `deploy.resources` limits. +- Container: Linux, port 8501, health check `curl http://localhost:8501/_stcore/health` (`Dockerfile`) +- Resource guidance from `docker-compose.yml`: NLTK app ~2G RAM; GPT-2 app ~4G; ensemble ~6G; GPU optional +- PaaS: `Procfile` supports Heroku-style deployment of the NLTK app +- CI: GitHub Actions (`.github/workflows/ci.yml`) - lint job on 3.11 (flake8/black/isort over `src/ tests/ app.py gpt2_app.py ensemble.py`) plus test matrix 3.9/3.10/3.11 with NLTK data download and coverage --- -*Stack analysis: 2026-04-02* +*Stack analysis: 2026-07-06* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md index 7786b1f..80b6aec 100644 --- a/.planning/codebase/STRUCTURE.md +++ b/.planning/codebase/STRUCTURE.md @@ -1,154 +1,69 @@ -# Codebase Structure +# Structure -**Analysis Date:** 2026-04-02 +**Analysis Date:** 2026-07-06 ## Directory Layout ``` AI_Text_Detector/ -├── app.py # Streamlit: NLTK-only detector UI -├── ensemble.py # Streamlit: ensemble (GPT-2 + NLTK, RoBERTa optional) -├── test.py # Streamlit: GPT-2-only UI (not pytest) -├── setup.py # setuptools: package_dir src, requirements.txt install_requires -├── requirements.txt # Runtime dependencies (referenced by setup.py) -├── docker-compose.yml # Container orchestration (not expanded here) -├── README.md # Project overview -├── .streamlit/ -│ └── config.toml # Streamlit theme/config -├── docs/ -│ ├── API.md # Programmatic analyzer usage examples -│ └── DEPLOYMENT.md # Deployment notes -├── tests/ -│ ├── conftest.py # pytest path fix + shared fixtures -│ ├── test_nltk_analyzer.py -│ ├── test_gpt2_analyzer.py -│ ├── test_roberta_analyzer.py -│ ├── test_ensemble_analyzer.py -│ ├── test_result_model.py -│ └── test_text_processing.py -└── src/ # Main Python package root (on sys.path in apps/tests) - ├── __init__.py - ├── analyzers/ - │ ├── __init__.py # Re-exports analyzer classes - │ ├── base_analyzer.py - │ ├── nltk_analyzer.py - │ ├── gpt2_analyzer.py - │ ├── roberta_analyzer.py - │ └── ensemble_analyzer.py - ├── models/ - │ ├── __init__.py - │ └── result.py # AnalysisResult, TextMetrics, DetectionScore - ├── config/ - │ ├── __init__.py - │ └── settings.py # Verdict, thresholds, get_settings() - └── utils/ - ├── __init__.py - ├── logging_config.py - ├── text_processing.py - └── visualization.py +├── app.py # Streamlit: NLTK-only UI (torch-free) +├── gpt2_app.py # Streamlit: GPT-2 UI (renamed from test.py) +├── ensemble.py # Streamlit: ensemble UI +├── src/ +│ ├── analyzers/ +│ │ ├── base_analyzer.py # Template Method: analyze() + _apply_input_cap +│ │ ├── nltk_analyzer.py # Brown-corpus n-gram (Witten-Bell smoothing, cached) +│ │ ├── gpt2_analyzer.py # GPT-2 perplexity (safetensors) +│ │ ├── roberta_analyzer.py # RoBERTa classifier (disabled by default) +│ │ ├── binoculars_analyzer.py # cross-perplexity (gpt2 + distilgpt2) +│ │ ├── calibration.py # logistic_ai_probability +│ │ ├── ensemble_analyzer.py # weighted calibrated fusion +│ │ ├── __init__.py # lazy __getattr__ for torch-backed analyzers +│ │ └── AGENTS.md +│ ├── config/settings.py # frozen dataclass configs + get_settings() singleton +│ ├── models/result.py # AnalysisResult / TextMetrics / DetectionScore +│ ├── ui/ # NEW: shared Streamlit components +│ │ ├── styles.py # BASE_CSS + inject_css +│ │ ├── components.py # render_verdict_card/warnings/footer/error, verdict maps +│ │ └── __init__.py +│ ├── evaluation/ # NEW: measurement layer +│ │ ├── metrics.py # pure-NumPy accuracy/PR/F1/ROC/AUROC/FPR/FNR/ECE +│ │ ├── dataset.py # JSONL loader +│ │ ├── benchmark.py # runner + CLI + plots +│ │ └── __init__.py +│ └── utils/ +│ ├── text_processing.py # TextProcessor (clean/tokenize/metrics/NLTK bootstrap) +│ ├── visualization.py # ChartGenerator (Plotly + Matplotlib Agg) +│ ├── ui_contract.py # shared UI copy (limitations, mode guidance) +│ ├── logging_config.py # setup_logging / get_logger +│ └── AGENTS.md +├── data/benchmark/ # samples.jsonl (12 human + 12 AI) + README +├── docs/ # API.md, DEPLOYMENT.md, benchmarks/ (reports + PNGs) +├── tests/ # 22 test files, ~231 test functions +├── .github/workflows/ci.yml # lint + test matrix +├── Dockerfile, docker-compose.yml, Procfile, Makefile +├── requirements.txt, requirements-dev.txt, setup.py, pytest.ini, .flake8 +└── README / CONTRIBUTING / SECURITY / CODE_OF_CONDUCT / CHANGELOG / LICENSE ``` -## Directory Purposes +## Key Locations -**Repository root:** -- Purpose: Operator entrypoints and packaging metadata. -- Contains: Streamlit scripts, `setup.py`, `requirements.txt`, compose file, top-level docs. -- Key files: `app.py`, `ensemble.py`, `test.py`, `setup.py` - -**`src/`:** -- Purpose: Installable library code; all business logic for detection and shared utilities. -- Contains: Analyzers, models, config, utils. -- Key files: `src/analyzers/base_analyzer.py`, `src/models/result.py`, `src/config/settings.py` - -**`src/analyzers/`:** -- Purpose: Pluggable detection backends and ensemble orchestration. -- Contains: One module per analyzer + package `__init__.py` exposing `__all__`. -- Key files: `src/analyzers/__init__.py`, `src/analyzers/ensemble_analyzer.py` - -**`src/models/`:** -- Purpose: Data transfer objects for analysis output. -- Contains: Dataclasses only in current tree. -- Key files: `src/models/result.py` - -**`src/config/`:** -- Purpose: Centralized enums and configuration objects. -- Key files: `src/config/settings.py` - -**`src/utils/`:** -- Purpose: Cross-cutting helpers (no UI). -- Key files: `src/utils/text_processing.py`, `src/utils/visualization.py`, `src/utils/logging_config.py` - -**`tests/`:** -- Purpose: Pytest suites mirroring analyzer and utility modules. -- Contains: `conftest.py` prepends `../src` to `sys.path` for imports like `src.analyzers...`. -- Key files: `tests/conftest.py`, `tests/test_ensemble_analyzer.py` - -**`docs/`:** -- Purpose: Human-facing API and deployment documentation. -- Key files: `docs/API.md`, `docs/DEPLOYMENT.md` - -## Key File Locations - -**Entry Points:** -- `app.py`: NLTK Streamlit application. -- `ensemble.py`: Ensemble Streamlit application. -- `test.py`: GPT-2 Streamlit application. - -**Configuration:** -- `src/config/settings.py`: Thresholds, NLTK/GPT-2/visualization knobs, `get_settings()`. -- `.streamlit/config.toml`: Streamlit UI configuration. - -**Core Logic:** -- `src/analyzers/base_analyzer.py`: Shared analysis pipeline and verdict logic. -- `src/analyzers/nltk_analyzer.py`, `gpt2_analyzer.py`, `roberta_analyzer.py`, `ensemble_analyzer.py`: Concrete detectors. - -**Testing:** -- `tests/*.py`: Mirror modules under `src/`; shared fixtures in `tests/conftest.py`. +- **Add an analyzer:** new file in `src/analyzers/`, extend `BaseAnalyzer`, + implement `_perform_analysis`, register in `__init__.py::_LAZY_MODULES` if it + needs torch. +- **Tune detection:** `src/config/settings.py` (thresholds, calibration + midpoints/slopes, ensemble weights). Never hardcode thresholds in analyzers. +- **Shared UI change:** `src/ui/` (once, not per app). +- **Evaluate:** `python -m src.evaluation.benchmark`; dataset in `data/benchmark/`. ## Naming Conventions -**Files:** -- Streamlit apps: short top-level names (`app.py`, `ensemble.py`) — except `test.py` which is a Streamlit GPT-2 app, not a pytest file. -- Library modules: `snake_case.py` under `src/`. -- Tests: `test_.py` in `tests/`. - -**Directories:** -- Package names: lowercase (`analyzers`, `utils`, `config`, `models`). - -**Classes:** -- Analyzers: `*Analyzer` suffix (`NLTKAnalyzer` in `src/analyzers/nltk_analyzer.py`). -- Utilities: `TextProcessor`, `ChartGenerator` in `src/utils/`. - -## Where to Add New Code - -**New detection backend:** -- Implementation: `src/analyzers/_analyzer.py` subclassing `BaseAnalyzer` from `src/analyzers/base_analyzer.py`. -- Registration: Export in `src/analyzers/__init__.py` `__all__`. -- Tests: `tests/test__analyzer.py`. -- Optional UI: New Streamlit file at repo root following `app.py` pattern (path insert + imports). - -**New feature on existing pipeline (e.g. extra metric):** -- Extend `AnalysisResult` / `TextMetrics` in `src/models/result.py` if the contract changes. -- Compute in `TextProcessor` or inside `_perform_analysis` depending on whether it is model-agnostic. -- Update `_determine_verdict` / `_generate_explanation` in `src/analyzers/base_analyzer.py` if verdict logic should use it globally. - -**New shared helper:** -- Add to `src/utils/` with a focused module name; import via `src.utils.`. - -**New Streamlit-only behavior:** -- Keep in the relevant root script (`app.py`, `ensemble.py`, `test.py`) or extract small pure functions into `src/utils/` if reused. - -## Special Directories - -**`.planning/codebase/`:** -- Purpose: GSD / planner-oriented codebase maps (this file and siblings). -- Generated: No — maintained by mapping workflow. -- Committed: Yes (typical for GSD projects). - -**`src/` as package root:** -- Purpose: `setup.py` uses `package_dir={"": "src"}` so installed import paths match development imports (`from src.analyzers...`). -- When running Streamlit from repo root, scripts also insert `src` into `sys.path` so the same `src.*` imports resolve. +- Modules `snake_case.py`; classes `PascalCase`; functions/vars `snake_case`. +- Tests `tests/test_.py`. Streamlit entries are short root scripts. +- Package `__init__.py` files exist under `src/`; import concrete symbols from + submodules. ---- +## Not Tracked (gitignored local artifacts) -*Structure analysis: 2026-04-02* +`venv/`, `graphify-out/`, `.obsidian/`, `.pytest_cache/`, `__pycache__/`, +`SESSION_AUDIT_LOG.md`, `/models/`, `*.safetensors`, `*.bin`. diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md index a6dfdcf..0347bd3 100644 --- a/.planning/codebase/TESTING.md +++ b/.planning/codebase/TESTING.md @@ -1,143 +1,64 @@ -# Testing Patterns +# Testing + +**Analysis Date:** 2026-07-06 + +## Framework & Layout + +- **pytest** (`requirements-dev.txt`, `setup.py extras_require[dev]`). +- `pytest.ini` registers the `slow` marker (`-m "not slow"` deselects + model-heavy tests). +- `tests/conftest.py` prepends `../src` to `sys.path` (no `PYTHONPATH` needed) + and provides shared text fixtures (`sample_ai_text`, `sample_human_text`, + `short_text`, `empty_text`, `medium_text`, `repetitive_text`). +- **22 test files, ~231 test functions.** Full-suite coverage ~**92%** of `src/`. + +## Test Areas + +- **Analyzer unit/contract:** `test_nltk_analyzer.py`, `test_gpt2_analyzer.py` + (slow, torch-gated), `test_roberta_analyzer.py`, `test_binoculars_analyzer.py` + (mocked + slow real-model), `test_base_analyzer_contract.py`, + `test_analyzer_internals.py` (pure verdict/explanation/interpret/input-cap/ + error-path — no model load). +- **Calibration/eval:** `test_calibration.py` (logistic mapping), + `test_evaluation.py`, `test_metrics`-style cases, `test_dataset_loader.py` + (error branches), `test_benchmark_runner.py` (factory, summary, plots, CLI). +- **Ensemble:** `test_ensemble_analyzer.py` (mocked sub-analyzers), + `test_ensemble_weighted_fusion.py` (calibrated fusion, binoculars gating, + human-scale-not-flagged regression). +- **UI:** `test_streamlit_apps.py` (Streamlit **AppTest** headless smoke tests + for all 3 entry scripts), `test_ui_components.py`, `test_ui_contract.py`, + `test_visualization.py` (ChartGenerator), `test_infra.py` (logging + lazy + imports), `test_*_streamlit_contract.py` (static source contracts). +- **Data model:** `test_result_model.py`. + +## Mocking Strategy + +- Sub-analyzers are replaced with lightweight fakes (`EnsembleAnalyzer`) or the + `_compute_*` method is monkeypatched (`BinocularsAnalyzer`) so most tests run + **without loading transformer or Brown-corpus models**. +- Genuinely model-dependent paths (`_compute_perplexity_gpt2`, + `_compute_binoculars`, real RoBERTa/GPT-2 `analyze`) are marked + `@pytest.mark.slow` and excluded from the default/CI run. +- `NLTKAnalyzer` uses a process-wide model cache, so the Brown model builds once + per test process (NLTK suite ~30s instead of ~6min). + +## Running -**Analysis Date:** 2026-04-02 - -## Test Framework - -**Runner:** -- **pytest** (>=7.4.0 per `requirements-dev.txt`). -- No committed `pytest.ini`, `pyproject.toml`, or `tox.ini` in repo root; behavior is default pytest discovery. - -**Assertion Library:** -- Plain `assert` statements (pytest style). - -**Plugins (declared):** -- `pytest-cov` — coverage. -- `pytest-mock` — available but **not used** in current test modules (no `mock` / `patch` / `mocker` references under `tests/`). - -**Run Commands:** -```bash -# Recommended (matches Makefile): PYTHONPATH set for src package -make test - -# Equivalent manual invocation from repo root -PYTHONPATH=src python -m pytest tests/ -v --cov=src --cov-report=html --cov-report=term-missing -``` - -On Windows PowerShell, set env then run pytest, or use `make test` if GNU Make is available. - -**README** also documents: `pytest tests/ -v --cov=src --cov-report=html` and single-file runs (e.g. `pytest tests/test_ensemble_analyzer.py -v`). - -## Test File Organization - -**Location:** -- All tests under `tests/` (not co-located with `src/`). - -**Naming:** -- `test_.py` (e.g. `tests/test_text_processing.py`, `tests/test_result_model.py`). - -**Package:** -- `tests/__init__.py` present (package layout). - -**Structure:** -``` -tests/ -├── __init__.py -├── conftest.py # shared fixtures, path bootstrap -├── test_ensemble_analyzer.py -├── test_gpt2_analyzer.py -├── test_nltk_analyzer.py -├── test_result_model.py -├── test_roberta_analyzer.py -└── test_text_processing.py -``` - -## Test Structure - -**Suite organization:** -- **Classes** group related cases: `TestTextCleaning`, `TestNLTKAnalyzerInit`, `TestEnsembleAnalyzer`, etc. - -**Example pattern (class + methods):** -```python -class TestTextCleaning: - def test_clean_empty_text(self): - assert TextProcessor.clean_text("") == "" -``` - -**Fixtures:** -- **Shared:** `tests/conftest.py` — `sample_ai_text`, `sample_human_text`, `short_text`, `empty_text`, `medium_text`, `repetitive_text`, `nltk_analyzer`, `text_processor`. -- **Local:** `@pytest.fixture` in module (e.g. `ensemble_analyzer` in `tests/test_ensemble_analyzer.py`). -- **Autouse setup:** `autouse=True` fixture on test class for analyzer instance (`tests/test_nltk_analyzer.py`, `tests/test_gpt2_analyzer.py`). - -**Parametrization:** Not heavily used; prefer explicit methods per edge case in current codebase. - -## Mocking - -**Framework:** `pytest-mock` is a dev dependency only; **current tests are integration-style** against real `NLTKAnalyzer`, `GPT2Analyzer`, `EnsembleAnalyzer`, and `TextProcessor`. - -**Prescriptive guidance:** -- Use `pytest-mock`’s `mocker` fixture or `unittest.mock.patch` for Hugging Face / torch downloads, slow I/O, or nondeterministic model output when adding CI-friendly unit tests. -- Keep heavy model tests behind markers (see below). - -## Fixtures and Factories - -**Test data:** -- Long representative strings in `conftest.py` for “AI-like” vs “human-like” prose and edge cases (short, empty, repetitive). - -**Factory-style fixtures:** -- `nltk_analyzer` → `NLTKAnalyzer(ngram_size=3)`. -- `text_processor` → `TextProcessor()`. - -**No separate `fixtures/` directory**; everything lives in `conftest.py` or inline fixtures. - -## Coverage - -**Requirements:** No enforced coverage threshold in repo config; `Makefile` `test` target runs coverage with HTML + terminal missing-line report. - -**View coverage:** ```bash -make test -# Open htmlcov/index.html after run +python -m pytest tests/ -m "not slow" -q # fast (CI default) +python -m pytest tests/ -q --cov=src --cov-report=term-missing # full + coverage +python -m pytest -m slow -v # model-backed only ``` -**Scope:** `--cov=src` limits reporting to the installable package under `src/`. - -## Test Types - -**Unit tests:** -- `tests/test_text_processing.py` — pure utilities, fast, no GPU. - -**Model / analyzer tests:** -- `tests/test_nltk_analyzer.py`, `tests/test_result_model.py` — logic close to domain. - -**Heavy / ML tests:** -- `tests/test_gpt2_analyzer.py` — module-level `pytestmark = pytest.mark.skipif(not HAS_TORCH, ...)`; class `TestGPT2Analysis` uses `@pytest.mark.slow` (documented for CI filtering in module docstring). -- `tests/test_roberta_analyzer.py` and `tests/test_ensemble_analyzer.py` exercise transformers stack (slow, network/model cache dependent). - -**E2E / UI:** Streamlit apps are not covered by automated browser tests in `tests/`. - -## Common Patterns - -**Enum membership:** -```python -assert result.verdict in list(Verdict) -``` - -**Exception testing:** -```python -with pytest.raises(ValueError): - analyzer.set_ngram_size(1) -``` - -**Structural assertions on results:** -- `result.to_dict()`, `hasattr` checks, score name substring checks (`"RoBERTa" in name`, etc.) in `tests/test_ensemble_analyzer.py`. - -## CI / Automation - -**Repository:** No `.github/workflows` or other CI config detected in workspace; quality gates are **local** via `Makefile` (`test`, `lint`, `format`). +## CI -**Pre-commit:** Listed in `requirements-dev.txt`; no committed hook config found — treat as optional local setup unless added later. +`.github/workflows/ci.yml`: **lint** job (flake8 + black --check + isort --check) +and **test** job over Python 3.9/3.10/3.11 — installs deps, downloads NLTK data, +runs `pytest -m "not slow" --cov=src`. ---- +## Gaps / Notes -*Testing analysis: 2026-04-02* +- Coverage is **not gated** (`--cov-fail-under` not set) — it can silently erode. +- Slow model tests **never run in CI** (no scheduled full-suite job). +- Bundled benchmark (`data/benchmark/samples.jsonl`) is 24 clean in-distribution + samples — used for regression/calibration, **not** an accuracy claim. From ef8a925c4bb16c7d25f6fe1d5fece20f21803cb5 Mon Sep 17 00:00:00 2001 From: Satyam Shivam Date: Tue, 7 Jul 2026 09:17:54 +0530 Subject: [PATCH 2/2] refactor: address score-ordering fragility (M2) and add CI coverage gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M2 — remove positional score coupling: The ensemble and Binoculars analyzers read the fused primary score via result.scores[0] and the voter set via scores[1:], so reordering or inserting a score would silently break verdict math. Add AnalysisResult.get_score(name) and address scores by name: - EnsembleAnalyzer.ENSEMBLE_SCORE_NAME constant + _voter_scores() helper. - BinocularsAnalyzer.PRIMARY_SCORE_NAME constant. New test asserts the ensemble verdict is unchanged when the primary score is moved out of index 0. CI — guard against coverage erosion: Add --cov-fail-under=80 to the test job. The fast (not-slow) suite currently covers 85% of src/, so this is a real floor without breaking the green build. Tests: 54 (M2 suites incl. real-model binoculars) + get_score unit tests; fast suite 231 passed at 85% coverage. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 2 +- src/analyzers/binoculars_analyzer.py | 10 +++++++--- src/analyzers/ensemble_analyzer.py | 24 ++++++++++++++++++------ src/models/result.py | 13 ++++++++++++- tests/test_ensemble_weighted_fusion.py | 22 ++++++++++++++++++++++ tests/test_result_model.py | 19 +++++++++++++++++++ 6 files changed, 79 insertions(+), 11 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c3b8316..d8f06db 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -51,4 +51,4 @@ jobs: - name: Download NLTK data run: python -c "import nltk; nltk.download(['punkt','punkt_tab','stopwords','brown','averaged_perceptron_tagger'])" - name: Run tests (excluding slow model tests) - run: python -m pytest tests/ -m "not slow" -q --cov=src --cov-report=term-missing + run: python -m pytest tests/ -m "not slow" -q --cov=src --cov-report=term-missing --cov-fail-under=80 diff --git a/src/analyzers/binoculars_analyzer.py b/src/analyzers/binoculars_analyzer.py index 5e6751f..9f482de 100644 --- a/src/analyzers/binoculars_analyzer.py +++ b/src/analyzers/binoculars_analyzer.py @@ -45,6 +45,9 @@ class BinocularsAnalyzer(BaseAnalyzer): """Cross-perplexity (two-model) AI-text detector.""" + # Primary calibrated score, addressed by name (not list position). + PRIMARY_SCORE_NAME = "Binoculars AI Score" + def __init__(self): super().__init__() self.config = self.settings.binoculars @@ -162,10 +165,10 @@ def _perform_analysis(self, text: str, result: AnalysisResult) -> AnalysisResult ) result.perplexity = observer_ppl - # Primary calibrated score (index 0, mirrors the ensemble contract). + # Primary calibrated score, addressed by name (not list position). result.add_score( DetectionScore( - name="Binoculars AI Score", + name=self.PRIMARY_SCORE_NAME, value=ai_prob, weight=1.0, interpretation=self._interpret(score, ai_prob), @@ -191,7 +194,8 @@ def _perform_analysis(self, text: str, result: AnalysisResult) -> AnalysisResult def _determine_verdict(self, result: AnalysisResult) -> AnalysisResult: """Map the calibrated Binoculars AI-probability to a verdict.""" - ai_prob = result.scores[0].value if result.scores else 0.5 + primary = result.get_score(self.PRIMARY_SCORE_NAME) + ai_prob = primary.value if primary else 0.5 confidence = abs(ai_prob - 0.5) * 200 # 0-100 if ai_prob >= 0.75: diff --git a/src/analyzers/ensemble_analyzer.py b/src/analyzers/ensemble_analyzer.py index f1cc44e..5e62df9 100644 --- a/src/analyzers/ensemble_analyzer.py +++ b/src/analyzers/ensemble_analyzer.py @@ -43,6 +43,10 @@ class EnsembleAnalyzer(BaseAnalyzer): noise until a fine-tuned checkpoint is wired in. """ + # Name of the fused primary score; addressed by name (not list position) so + # the order scores are added in is not a load-bearing contract. + ENSEMBLE_SCORE_NAME = "Ensemble AI Score" + def __init__(self): super().__init__() self.method_name = "Ensemble (GPT2+NLTK)" @@ -271,7 +275,7 @@ def _combine_results( # Add ensemble score (always index 0 by contract). result.add_score( DetectionScore( - name="Ensemble AI Score", + name=self.ENSEMBLE_SCORE_NAME, value=ensemble_ai_score, weight=1.0, interpretation=self._interpret_ensemble_score(ensemble_ai_score), @@ -341,6 +345,12 @@ def _combine_results( return result + def _voter_scores(self, result: AnalysisResult) -> list: + """Contributing analyzer scores: everything except the fused primary + score, restricted to non-zero weight (so a disabled analyzer's row is + never counted as a voter).""" + return [s for s in result.scores if s.name != self.ENSEMBLE_SCORE_NAME and s.weight > 0] + def _determine_verdict(self, result: AnalysisResult) -> AnalysisResult: """ Determine final verdict based on ensemble score. @@ -353,12 +363,13 @@ def _determine_verdict(self, result: AnalysisResult) -> AnalysisResult: """ cfg = self.ensemble_config - # Get ensemble AI score (index 0 by contract). - ensemble_score = result.scores[0].value if result.scores else 0.5 + # Get the fused score by name (not list position). + primary = result.get_score(self.ENSEMBLE_SCORE_NAME) + ensemble_score = primary.value if primary else 0.5 # Agreement only over analyzers that actually contribute (weight > 0): # a disabled/zero-weight RoBERTa must not sway confidence or narrative. - voters = [s for s in result.scores[1:] if s.weight > 0] + voters = self._voter_scores(result) ai_votes = sum(1 for s in voters if s.indicates_ai) total_votes = len(voters) agreement = ai_votes / total_votes if total_votes > 0 else 0.5 @@ -425,7 +436,8 @@ def _generate_ensemble_explanation( parts = [] # Ensemble verdict - ensemble_score = result.scores[0].value if result.scores else 0.5 + primary = result.get_score(self.ENSEMBLE_SCORE_NAME) + ensemble_score = primary.value if primary else 0.5 parts.append( f"🎯 **Ensemble Analysis**: Combined score of {ensemble_score:.1%} " f"indicates **{result.verdict.value}** with {result.confidence:.1f}% confidence." @@ -454,7 +466,7 @@ def _generate_ensemble_explanation( ) # Agreement analysis over contributing analyzers only (weight > 0). - voters = [s for s in result.scores[1:] if s.weight > 0] + voters = self._voter_scores(result) ai_votes = sum(1 for s in voters if s.indicates_ai) total_votes = len(voters) if total_votes and ai_votes == total_votes: diff --git a/src/models/result.py b/src/models/result.py index 110700e..69b5ef1 100644 --- a/src/models/result.py +++ b/src/models/result.py @@ -10,7 +10,7 @@ import json from dataclasses import asdict, dataclass, field from datetime import datetime -from typing import Dict, List +from typing import Dict, List, Optional from src.config.settings import ConfidenceLevel, Verdict @@ -107,6 +107,17 @@ def add_score(self, score: DetectionScore) -> None: """Add a detection score.""" self.scores.append(score) + def get_score(self, name: str) -> Optional[DetectionScore]: + """Return the first score with the given name, or None. + + Lets callers address scores by name instead of list position, so the + order in which scores are added is not a load-bearing contract. + """ + for score in self.scores: + if score.name == name: + return score + return None + def to_dict(self) -> Dict: """Convert to dictionary for serialization.""" return { diff --git a/tests/test_ensemble_weighted_fusion.py b/tests/test_ensemble_weighted_fusion.py index dbbdcb3..dcd0702 100644 --- a/tests/test_ensemble_weighted_fusion.py +++ b/tests/test_ensemble_weighted_fusion.py @@ -106,6 +106,28 @@ def test_binoculars_contributes_when_weighted(): assert ensemble_score >= 0.3 * 0.9 +def test_verdict_is_robust_to_score_reordering(): + """The fused score is addressed by name, so inserting scores before it (or + reordering) must not change the verdict — guards the old scores[0] contract.""" + analyzer = EnsembleAnalyzer() + base_result = AnalysisResult(metrics=TextMetrics(total_words=40, unique_words=30)) + roberta_result = analyzer._disabled_roberta_result() + gpt2_result = AnalysisResult(verdict=Verdict.LIKELY_AI, perplexity=15.0) + nltk_result = AnalysisResult(verdict=Verdict.LIKELY_AI, perplexity=3000.0) + + combined = analyzer._combine_results(base_result, roberta_result, gpt2_result, nltk_result) + analyzer._determine_verdict(combined) + verdict_before = combined.verdict + + # Move the primary "Ensemble AI Score" out of index 0. + primary = combined.get_score(analyzer.ENSEMBLE_SCORE_NAME) + combined.scores.remove(primary) + combined.scores.append(primary) + analyzer._determine_verdict(combined) + + assert combined.verdict == verdict_before + + def test_disabled_roberta_excluded_from_agreement(): analyzer = EnsembleAnalyzer() base_result = AnalysisResult(metrics=TextMetrics(total_words=40, unique_words=30)) diff --git a/tests/test_result_model.py b/tests/test_result_model.py index 1186646..f3c82ae 100644 --- a/tests/test_result_model.py +++ b/tests/test_result_model.py @@ -8,6 +8,25 @@ from src.models.result import AnalysisResult, DetectionScore, TextMetrics +class TestGetScore: + """Tests for AnalysisResult.get_score name-based lookup.""" + + def test_returns_matching_score(self): + result = AnalysisResult() + result.add_score(DetectionScore(name="A", value=0.1)) + result.add_score(DetectionScore(name="B", value=0.9)) + assert result.get_score("B").value == 0.9 + + def test_returns_none_when_absent(self): + assert AnalysisResult().get_score("nope") is None + + def test_returns_first_on_duplicate_names(self): + result = AnalysisResult() + result.add_score(DetectionScore(name="X", value=0.2)) + result.add_score(DetectionScore(name="X", value=0.7)) + assert result.get_score("X").value == 0.2 + + class TestTextMetrics: """Tests for TextMetrics model."""