Skip to content

chore: post-audit quick wins (name-based scores, CI coverage gate) #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
satyamshivam13 merged 2 commits into from
Jul 8, 2026
Merged

chore: post-audit quick wins (name-based scores, CI coverage gate) #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cd011a9
docs: refresh codebase map to post-remediation state
satyamshivam13 Jul 7, 2026
ef8a925
refactor: address score-ordering fragility (M2) and add CI coverage gate
satyamshivam13 Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,4 @@ jobs:
- name: Download NLTK data
run: python -c "import nltk; nltk.download(['punkt','punkt_tab','stopwords','brown','averaged_perceptron_tagger'])"
- name: Run tests (excluding slow model tests)
run: python -m pytest tests/ -m "not slow" -q --cov=src --cov-report=term-missing
run: python -m pytest tests/ -m "not slow" -q --cov=src --cov-report=term-missing --cov-fail-under=80
171 changes: 54 additions & 117 deletions .planning/codebase/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,136 +1,73 @@
# Architecture

**Analysis Date:** 2026-04-02
**Analysis Date:** 2026-07-06

## Pattern Overview

**Overall:** Layered monolith with a **Template Method** analyzer hierarchy, optional **ensemble composition**, and **Streamlit scripts as thin presentation shells**.

**Key Characteristics:**
- Core detection logic lives under `src/` as importable packages; runnable UIs prepend `src` to `sys.path` then import `src.*` explicitly.
- `BaseAnalyzer` centralizes validation, metric computation, verdict scoring, explanations, and timing; concrete analyzers only implement `_perform_analysis`.
- `EnsembleAnalyzer` is a special case: it **overrides** `analyze()` to run multiple backends and fuse outputs instead of using the base `_perform_analysis` hook alone.
- Shared preprocessing and NLTK bootstrap live in `TextProcessor` (`src/utils/text_processing.py`); charts in `ChartGenerator` (`src/utils/visualization.py`).
Layered Python package (`src/`) behind three thin Streamlit UIs. Core detection
uses the **Template Method** pattern: `BaseAnalyzer.analyze()` is concrete and
orchestrates the pipeline; subclasses implement only `_perform_analysis()`.
`EnsembleAnalyzer` is the exception — it **overrides `analyze()`** to run and fuse
multiple sub-analyzers.

## Layers

**Presentation (Streamlit):**
- Purpose: Page config, layout, CSS, widgets, call into analyzers and charts, display `AnalysisResult`.
- Location: repository root — `app.py`, `ensemble.py`, `test.py` (GPT-2 UI; filename is legacy).
- Contains: Procedural Streamlit code, `@st.cache_resource` loaders, verdict-to-CSS mapping.
- Depends on: `src.analyzers.*`, `src.config.settings`, `src.utils.logging_config`, `src.utils.visualization`.
- Used by: End users via `streamlit run <file>.py`.

**Analysis (domain services):**
- Purpose: Turn raw text into `AnalysisResult` (verdict, scores, metrics, warnings).
- Location: `src/analyzers/`
- Contains: `BaseAnalyzer`, `NLTKAnalyzer`, `GPT2Analyzer`, `RoBERTaAnalyzer`, `EnsembleAnalyzer`.
- Depends on: `src.config.settings`, `src.models.result`, `src.utils.text_processing`, `src.utils.logging_config`; transformers stack in GPT-2/RoBERTa paths.
- Used by: Streamlit apps, tests, any programmatic `analyzer.analyze(text)` caller.

**Models:**
- Purpose: Serializable result shapes (`AnalysisResult`, `TextMetrics`, `DetectionScore`).
- Location: `src/models/result.py`
- Contains: `@dataclass` types, `to_dict()` / `to_json()` on `AnalysisResult`.
- Depends on: `Verdict`, `ConfidenceLevel` from `src/config/settings.py`.
- Used by: All analyzers and UI layers that render or export results.

**Configuration:**
- Purpose: Enums, frozen threshold and subsystem configs, cached `Settings` singleton.
- Location: `src/config/settings.py`
- Contains: `Verdict`, `ConfidenceLevel`, `ThresholdConfig`, `NLTKConfig`, `GPT2Config`, `VisualizationConfig`, `Settings`, `get_settings()` (`@lru_cache`).
- Depends on: `os.environ` for `AI_DETECTOR_DEBUG`, `AI_DETECTOR_LOG_LEVEL`.
- Used by: `BaseAnalyzer`, `ChartGenerator`, apps.

**Infrastructure utilities:**
- Purpose: Logging setup, text cleaning/tokenization/metrics, optional NLTK downloads, plotting.
- Location: `src/utils/logging_config.py`, `src/utils/text_processing.py`, `src/utils/visualization.py`
- Contains: `TextProcessor` (classmethods for corpus-safe NLP), `ChartGenerator` (Plotly primary, Matplotlib `Agg` backend).
- Depends on: NLTK, Plotly, Matplotlib, NumPy; `get_settings()` where needed.
- Used by: Analyzers and Streamlit tabs.
1. **Presentation** — `app.py` (NLTK), `gpt2_app.py` (GPT-2; renamed from
`test.py`), `ensemble.py`. Each prepends `src/` to `sys.path`, configures the
page, injects shared CSS, wires a specific analyzer, and renders an
`AnalysisResult`. Shared UI lives in **`src/ui/`** (styles + components).
2. **Analyzers** — `src/analyzers/`: `BaseAnalyzer`, `NLTKAnalyzer`,
`GPT2Analyzer`, `RoBERTaAnalyzer`, `BinocularsAnalyzer`, `EnsembleAnalyzer`,
plus `calibration.py` (perplexity → AI-probability logistic).
3. **Models/Contract** — `src/models/result.py`: `AnalysisResult`, `TextMetrics`,
`DetectionScore` dataclasses with `to_dict()`/`to_json()`.
4. **Config** — `src/config/settings.py`: frozen dataclasses (`ThresholdConfig`,
`NLTKConfig`, `GPT2Config`, `RoBERTaConfig`, `BinocularsConfig`,
`EnsembleConfig`, `VisualizationConfig`) + `get_settings()` `@lru_cache`
singleton; enums `Verdict`, `ConfidenceLevel`, `DetectionMethod`.
5. **Utils** — `src/utils/`: `text_processing.py` (`TextProcessor`),
`visualization.py` (`ChartGenerator`), `ui_contract.py` (shared UI copy),
`logging_config.py`.
6. **Evaluation** — `src/evaluation/`: `metrics.py` (pure-NumPy accuracy/PR/F1/
ROC-AUROC/FPR-FNR/ECE), `dataset.py` (JSONL loader over `data/benchmark/`),
`benchmark.py` (runner + `--analyzer {nltk,gpt2,binoculars,ensemble}` CLI +
ROC/calibration plots).

## Data Flow

**Single-analyzer (NLTK / GPT-2) path:**

1. UI collects `text` and calls `analyzer.analyze(text)` (e.g. `NLTKAnalyzer` from `app.py` via `load_analyzer`).
2. `BaseAnalyzer.analyze()` in `src/analyzers/base_analyzer.py` cleans with `TextProcessor.clean_text()`, sets `text_length`, handles empty/short input warnings.
3. `TextProcessor.compute_metrics()` fills `result.metrics` (`TextMetrics`).
4. Subclass `_perform_analysis()` computes model-specific signals and populates perplexity, burstiness, lexical diversity, sentence variance, and `DetectionScore` entries.
5. `_determine_verdict()` maps weighted feature scores to `Verdict` and `ConfidenceLevel`; `_generate_explanation()` builds the narrative string.
6. UI reads `AnalysisResult` fields and passes word frequencies / scores to `ChartGenerator` for Plotly figures.

**Ensemble path:**

1. `EnsembleAnalyzer.analyze()` in `src/analyzers/ensemble_analyzer.py` duplicates early validation/metrics steps then lazy-loads `RoBERTaAnalyzer`, `GPT2Analyzer`, `NLTKAnalyzer` via properties.
2. Component analyzers run (RoBERTa weight may be zeroed by default); scores are fused using configured weights.
3. Returns a single `AnalysisResult` consistent with the shared model type.

**State Management:**
- No server-side session store; Streamlit reruns the script. Heavy objects (analyzer, `ChartGenerator`) are cached with `@st.cache_resource` in each app file.
- Settings are process-wide via `get_settings()` cache.
`text → TextProcessor.clean_text → _apply_input_cap (max 50k chars) →
compute_metrics → _perform_analysis (per analyzer) → _determine_verdict →
_generate_explanation → AnalysisResult`. The ensemble instead runs GPT-2 + NLTK
(+ optionally RoBERTa/Binoculars when weighted), maps each to a **calibrated**
AI-probability via `logistic_ai_probability` (per-analyzer midpoint = decision
boundary), and fuses them by weight (clamped to `[0,1]`).

## Key Abstractions

**`BaseAnalyzer`:**
- Purpose: Uniform pipeline for text-in → `AnalysisResult` out.
- Examples: `src/analyzers/base_analyzer.py`
- Pattern: Template Method — `analyze()` is concrete; `_perform_analysis()` abstract.

**`TextProcessor`:**
- Purpose: Single place for NLTK data guarantees, tokenization, and `TextMetrics` computation.
- Examples: `src/utils/text_processing.py`
- Pattern: Stateful class with class-level caches (`_nltk_initialized`, stopwords).

**`AnalysisResult` / `DetectionScore`:**
- Purpose: Stable contract between analysis and presentation/API docs.
- Examples: `src/models/result.py`
- Pattern: Dataclasses with helper methods (`add_warning`, `add_score`, `to_dict`).

**`ChartGenerator`:**
- Purpose: Decouple Plotly/Matplotlib construction from Streamlit layout.
- Examples: `src/utils/visualization.py`
- Pattern: Small service class configured from `VisualizationConfig`.
- **BaseAnalyzer** (Template Method) — one abstract hook, shared validation/
scoring/explanation/timing; input-size cap in `_apply_input_cap`.
- **AnalysisResult** — stable serializable contract between analysis and UI/eval.
- **EnsembleConfig calibration** — decision boundaries and fusion weights live in
config, derived empirically from benchmark perplexity separation (GPT-2
0.75 / NLTK 0.25; RoBERTa & Binoculars 0.0, gated).
- **Lazy transformer imports** — `src/analyzers/__init__.py::__getattr__` defers
`GPT2Analyzer`/`RoBERTaAnalyzer`/`BinocularsAnalyzer`/`EnsembleAnalyzer` so
`app.py` runs without `torch`.
- **Shared UI** — `src/ui/components.py` (`render_verdict_card`,
`render_warnings`, `render_footer`, `render_error`, verdict maps; HTML-escaped)
and `src/ui/styles.py` (`BASE_CSS` + `inject_css`).

## Entry Points

**NLTK Streamlit app:**
- Location: `app.py`
- Triggers: `streamlit run app.py`
- Responsibilities: NLTK-only UX; `load_analyzer(ngram_size)` → `NLTKAnalyzer`.

**Ensemble Streamlit app:**
- Location: `ensemble.py`
- Triggers: `streamlit run ensemble.py`
- Responsibilities: `EnsembleAnalyzer` UX; combined GPT-2 + NLTK (+ optional RoBERTa).

**GPT-2 Streamlit app:**
- Location: `test.py`
- Triggers: `streamlit run test.py`
- Responsibilities: `GPT2Analyzer`-only UX (despite the name `test.py`).

**Library / programmatic use:**
- Documented in `docs/API.md`; import analyzers from `src.analyzers` and call `.analyze(text)`.

**Packaging:**
- Location: `setup.py` — `packages=find_packages(where="src")`, `package_dir={"": "src"}` so the installable name maps to the `src` tree.

## Error Handling

**Strategy:** Defensive defaults inside `BaseAnalyzer.analyze()`: exceptions in the try-block set `Verdict.UNCERTAIN`, zero confidence, append warning, generic explanation; logged with `logger.error(..., exc_info=True)`. Streamlit layers wrap display paths in `try/except` and call `st.error`.

**Patterns:**
- Empty/short text: early return with warnings, no model call.
- NLTK corpus missing: `NLTKAnalyzer._build_model` raises `RuntimeError` with download hint after logging.

## Cross-Cutting Concerns

**Logging:** `setup_logging` / `get_logger` from `src/utils/logging_config.py` at app startup and in analyzers.

**Validation:** Length thresholds from `ThresholdConfig` in `src/config/settings.py`; cleaning in `TextProcessor.clean_text()`.

**Authentication:** Not applicable — local Streamlit apps, no auth layer in repo.
- `streamlit run app.py` → `NLTKAnalyzer` (lightest, torch-free).
- `streamlit run gpt2_app.py` → `GPT2Analyzer`.
- `streamlit run ensemble.py` → `EnsembleAnalyzer`.
- `python -m src.evaluation.benchmark ...` → benchmark CLI.
- Programmatic: `from src.analyzers.<x> import <Analyzer>; analyzer.analyze(text)`.

---
## Error Handling Contract

*Architecture analysis: 2026-04-02*
All analysis exceptions are caught inside `analyze()`: set `Verdict.UNCERTAIN`,
zero confidence, log full traceback (`exc_info=True`), and append a **generic**
warning (no exception strings). UIs render errors via `src.ui.render_error`
(generic message to user, full trace to server log).
Loading
Loading