diff --git a/docs/INTERNALS.md b/docs/INTERNALS.md index c81e05ee..47635ab5 100644 --- a/docs/INTERNALS.md +++ b/docs/INTERNALS.md @@ -254,7 +254,7 @@ Reason: [Concise reason why this score was given.] ### Refinement Trigger -Sections scoring below the threshold (default: 80) trigger additional research: +Sections scoring below the threshold (default: 70) trigger additional research: ```python if score < GRADE_THRESHOLD: @@ -339,29 +339,28 @@ Phase 1: Deep Research (Lead Researcher) - Role: Gather facts, data, citations - Output: ~12 page research dossier -Phase 2: Section Writing (Gemini 3 Flash) - - Model: gemini-3-flash-preview +Phase 2: Section Writing (Gemini 3.1 Pro) + - Model: gemini-3.1-pro-preview - Role: Write each section with analytical depth - - Input: Dossier + previous sections in prompt + - Input: Dossier via previous_interaction_id + prior sections - Output: 20 sections × ~1.5 pages = 30+ pages ``` -This architecture treats Deep Research as the **researcher** and Gemini 3 Flash as the **writer**. +This architecture treats Deep Research as the **researcher** and Gemini 3.1 Pro as the **writer**. -### Why Not `previous_interaction_id`? +### Section Writing and `previous_interaction_id` -The Gemini docs mention `previous_interaction_id` for follow-up questions. However, testing showed that direct `generate_content()` calls with the dossier in the prompt produce better results: - -- More reliable (no interaction state to manage) -- Faster (no interaction lookup overhead) -- More controllable (full prompt visible) +Section writing continues the Deep Research interaction through +`previous_interaction_id`, so each section builds statefully on the researched +dossier instead of re-sending the full dossier in every prompt. See +[ARCHITECTURE](ARCHITECTURE.md) for the current end-to-end pipeline diagram. ### Model Selection | Component | Model | Rationale | |-----------|-------|-----------| | Research Dossier | `deep-research-preview-04-2026` | Autonomous web research | -| Section Writing | `gemini-3-flash-preview` | Fast, intelligent, cost-effective | +| Section Writing | `gemini-3.1-pro-preview` | Analytical depth, tiered pricing | | Stage 1 Analysis | `gemini-3-flash-preview` | Quick section analysis | ### Adaptive Polling diff --git a/docs/README.md b/docs/README.md index 3c8c2297..08598eeb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,61 +11,68 @@ Guides are grouped by what you are trying to do, following the [Diataxis](https://diataxis.fr/) split: learning, doing, looking up, and understanding. +> **Index currency:** reviewed against primr **1.37.1** on **2026-07-21**. The +> *Updated* column is each document's last substantive revision (git +> `last-commit` date). `tests/test_docs_index.py` fails CI if a doc under +> `docs/` is missing from this map or an index link does not resolve, so the +> map and the tree cannot silently drift apart. + ## Getting started (tutorials) -| Document | Description | -|----------|-------------| -| [ZERO_COST](ZERO_COST.md) | Run keyless collection and finish a sourced dossier with an existing agent plan | -| [API_KEYS](API_KEYS.md) | API key setup, security, and troubleshooting | -| [CONFIG](CONFIG.md) | First-run configuration and the full settings reference | -| [RUN_MODES](RUN_MODES.md) | Run modes, costs, strategy selection, output locations, and zero-cost `primr render` (Markdown to DOCX/TXT) | -| [AZURE_QUICKSTART](AZURE_QUICKSTART.md) | Stand up the team/org Azure deployment end to end | +| Document | Description | Updated | +|----------|-------------|---------| +| [ZERO_COST](ZERO_COST.md) | Run keyless collection and finish a sourced dossier with an existing agent plan | 2026-07-18 | +| [API_KEYS](API_KEYS.md) | API key setup, validation (`primr keys test`), security, and troubleshooting | 2026-07-21 | +| [CONFIG](CONFIG.md) | First-run configuration and the full settings reference | 2026-07-21 | +| [RUN_MODES](RUN_MODES.md) | Run modes, costs, strategy selection, output locations, and zero-cost `primr render` (Markdown to DOCX/TXT) | 2026-07-21 | +| [AZURE_QUICKSTART](AZURE_QUICKSTART.md) | Stand up the team/org Azure deployment end to end | 2026-07-18 | ## How-to guides (task-oriented) -| Document | Description | -|----------|-------------| -| [BATCH](BATCH.md) | Run primr across many companies from a CSV | -| [RECOVERY](RECOVERY.md) | Resume after a crash, reboot, or interrupted run | -| [IMPROVE](IMPROVE.md) | Improve and refine an existing report (`primr improve` / `refine`) | -| [SKILL_PACK](SKILL_PACK.md) | `primr skills` end to end: planning, curation, artifacts, CLI/MCP | -| [EVAL](EVAL.md) | Evaluate and compare models with the eval harness | -| [MODEL_ONBOARDING](MODEL_ONBOARDING.md) | Register and validate a new model | -| [AGENT_INTEGRATION](AGENT_INTEGRATION.md) | Operate Primr from MCP, A2A, skills, and agent hosts | -| [OPENCLAW](OPENCLAW.md) | OpenClaw integration and governed workflows | -| [COPILOT_COWORK_GUIDE](COPILOT_COWORK_GUIDE.md) | Sideload a skill pack into Microsoft 365 Copilot Cowork | -| [COPILOT_STUDIO_GUIDE](COPILOT_STUDIO_GUIDE.md) | Use primr from Copilot Studio | -| [FOUNDRY_AGENT_GUIDE](FOUNDRY_AGENT_GUIDE.md) | Use primr from a Microsoft Foundry agent | -| [CLOUD_DEPLOYMENT](CLOUD_DEPLOYMENT.md) | Serverless deployment on AWS, Azure, and GCP | -| [SECURITY_OPS](SECURITY_OPS.md) | Key rotation, audit logs, operational security | +| Document | Description | Updated | +|----------|-------------|---------| +| [BATCH](BATCH.md) | Run primr across many companies from a CSV | 2026-07-18 | +| [RECOVERY](RECOVERY.md) | Resume after a crash, reboot, or interrupted run | 2026-07-17 | +| [IMPROVE](IMPROVE.md) | Improve and refine an existing report (`primr improve` / `refine`) | 2026-04-10 | +| [SKILL_PACK](SKILL_PACK.md) | `primr skills` end to end: planning, curation, artifacts, CLI/MCP | 2026-07-17 | +| [EVAL](EVAL.md) | Evaluate and compare models with the eval harness | 2026-07-17 | +| [MODEL_ONBOARDING](MODEL_ONBOARDING.md) | Register and validate a new model | 2026-06-29 | +| [AGENT_INTEGRATION](AGENT_INTEGRATION.md) | Operate Primr from MCP, A2A, skills, and agent hosts | 2026-07-21 | +| [OPENCLAW](OPENCLAW.md) | OpenClaw integration and governed workflows | 2026-07-17 | +| [COPILOT_COWORK_GUIDE](COPILOT_COWORK_GUIDE.md) | Sideload a skill pack into Microsoft 365 Copilot Cowork | 2026-07-21 | +| [COPILOT_STUDIO_GUIDE](COPILOT_STUDIO_GUIDE.md) | Use primr from Copilot Studio | 2026-07-21 | +| [FOUNDRY_AGENT_GUIDE](FOUNDRY_AGENT_GUIDE.md) | Use primr from a Microsoft Foundry agent | 2026-07-21 | +| [CLOUD_DEPLOYMENT](CLOUD_DEPLOYMENT.md) | Serverless deployment on AWS, Azure, and GCP | 2026-07-18 | +| [SECURITY_OPS](SECURITY_OPS.md) | Key rotation, audit logs, operational security | 2026-07-17 | ## Reference (look-up) -| Document | Description | -|----------|-------------| -| [API](API.md) | MCP server and A2A protocol, programmatic usage | -| [Job Status](JOB_STATUS.md) | Versioned CLI, MCP, A2A, and API lifecycle contract | -| [STRATEGY_PORTFOLIO](STRATEGY_PORTFOLIO.md) | Strategy document types and selection | -| [NEXT_STEPS](NEXT_STEPS.md) | What to build next, why it comes next, and what not to do yet | -| [CHANGELOG](CHANGELOG.md) | Version history | -| [MIGRATION](MIGRATION.md) | Error-hierarchy migration notes | -| [ROADMAP](https://github.com/blisspixel/primr/blob/main/ROADMAP.md) | Ordered development queue and version plan | +| Document | Description | Updated | +|----------|-------------|---------| +| [API](API.md) | MCP server and A2A protocol, programmatic usage | 2026-07-18 | +| [Job Status](JOB_STATUS.md) | Versioned CLI, MCP, A2A, and API lifecycle contract | 2026-07-10 | +| [STRATEGY_PORTFOLIO](STRATEGY_PORTFOLIO.md) | Strategy document types and selection | 2026-07-21 | +| [NEXT_STEPS](NEXT_STEPS.md) | What to build next, why it comes next, and what not to do yet | 2026-07-21 | +| [CHANGELOG](CHANGELOG.md) | Version history | 2026-07-21 | +| [MIGRATION](MIGRATION.md) | Error-hierarchy migration notes | 2026-02-02 | +| [EVAL_V1_24_0](EVAL_V1_24_0.md) | Historical decision record: the v1.24.0 cross-provider eval plan | 2026-06-26 | +| [ROADMAP](https://github.com/blisspixel/primr/blob/main/ROADMAP.md) | Ordered development queue and version plan | — | ## Explanation (understanding why) -| Document | Description | -|----------|-------------| -| [ARCHITECTURE](ARCHITECTURE.md) | System design and the 9-tier scraping engine | -| [ARTIFACTS](ARTIFACTS.md) | The research-vs-shipping artifact pipeline and ship-time gates | -| [INTERNALS](INTERNALS.md) | Core algorithms and prompt strategy | -| [STATE_MACHINES](STATE_MACHINES.md) | Tier escalation and job lifecycle | -| [CONCURRENCY](CONCURRENCY.md) | Threading and concurrency model | -| [SECURITY](SECURITY.md) | Security policy and the scoped AI/agent threat model | -| [design/](design/README.md) | Per-workstream design docs and decision audits (incl. the v1.24.0 cross-provider eval) | +| Document | Description | Updated | +|----------|-------------|---------| +| [ARCHITECTURE](ARCHITECTURE.md) | System design and the 9-tier scraping engine | 2026-07-21 | +| [ARTIFACTS](ARTIFACTS.md) | The research-vs-shipping artifact pipeline and ship-time gates | 2026-07-21 | +| [INTERNALS](INTERNALS.md) | Core algorithms and prompt strategy | 2026-07-21 | +| [STATE_MACHINES](STATE_MACHINES.md) | Tier escalation and job lifecycle | 2026-02-02 | +| [CONCURRENCY](CONCURRENCY.md) | Threading and concurrency model | 2026-07-12 | +| [SECURITY](SECURITY.md) | Security policy and the scoped AI/agent threat model | 2026-07-21 | +| [design/](design/README.md) | Per-workstream design docs and decision audits | — | ## Contributing -| Document | Description | -|----------|-------------| -| [CONTRIBUTING](CONTRIBUTING.md) | Dev environment setup and the contribution workflow | -| [CLAUDE.md](https://github.com/blisspixel/primr/blob/main/CLAUDE.md) | The development contract: seams, constraints, verification gates | +| Document | Description | Updated | +|----------|-------------|---------| +| [CONTRIBUTING](CONTRIBUTING.md) | Dev environment setup and the contribution workflow | 2026-07-09 | +| [CLAUDE.md](https://github.com/blisspixel/primr/blob/main/CLAUDE.md) | The development contract: seams, constraints, verification gates | — | diff --git a/tests/test_docs_index.py b/tests/test_docs_index.py new file mode 100644 index 00000000..d1da8c9b --- /dev/null +++ b/tests/test_docs_index.py @@ -0,0 +1,48 @@ +"""The docs index (`docs/README.md`) must stay in lockstep with the doc tree. + +Two failure modes rot a documentation map: a new doc lands but never gets +listed (readers cannot find it), or an entry outlives the file it points to +(a dead link). Both are caught here without needing git history, so the check +is safe under a shallow CI checkout. + +The `Updated` column dates in the index are informational (git last-commit +dates, refreshed on review) and are deliberately not asserted here -- pinning +exact dates would be brittle across rebases and squash merges. What must never +drift is *coverage* and *link validity*, which is what this test guards. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +DOCS_DIR = REPO_ROOT / "docs" +INDEX = DOCS_DIR / "README.md" + +_LINK = re.compile(r"\]\(([^)]+)\)") + + +def _index_link_targets() -> list[str]: + return _LINK.findall(INDEX.read_text(encoding="utf-8")) + + +def test_every_top_level_doc_is_listed_in_the_index() -> None: + targets = {Path(t).name for t in _index_link_targets()} + missing = sorted( + path.name + for path in DOCS_DIR.glob("*.md") + if path.name != "README.md" and path.name not in targets + ) + assert not missing, f"docs/README.md must link every doc under docs/. Missing: {missing}" + + +def test_every_relative_index_link_resolves() -> None: + broken: list[str] = [] + for target in _index_link_targets(): + if target.startswith(("http://", "https://", "#", "mailto:")): + continue + resolved = (DOCS_DIR / target.split("#", 1)[0]).resolve() + if not resolved.exists(): + broken.append(target) + assert not broken, f"docs/README.md has links that do not resolve: {broken}"