Skip to content

feat: plan annotations, code review mode, E2E test scenarios in spec workflow#120

Closed
maxritter wants to merge 1 commit intomainfrom
dev
Closed

feat: plan annotations, code review mode, E2E test scenarios in spec workflow#120
maxritter wants to merge 1 commit intomainfrom
dev

Conversation

@maxritter
Copy link
Copy Markdown
Owner

@maxritter maxritter commented Mar 27, 2026

See commit message for full changelog.

Summary by CodeRabbit

Release Notes

  • New Features

    • Added Headless Mode (pilot -p) for non-interactive CLI runs with full flag compatibility
    • Plan Annotation feature in Console Specifications tab for visual markup and notes
    • Code Review mode in Console Changes tab with inline diff-line annotations
    • Enterprise tier with full source code access, forking, and modification rights (100+ seats)
    • Free 7-day trial for all plans (no credit card required)
  • Documentation

    • Updated CLI documentation with headless mode examples and usage patterns
    • Added Enterprise licensing tier and subscription details
    • Updated installation guide reflecting browser automation tooling changes
  • Chores

    • Replaced browser automation tooling dependency for improved performance

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 27, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
pilot-shell Ignored Ignored Preview Mar 27, 2026 8:47am

Request Review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 27, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: da11f778-9765-4fc2-a94c-29664090caf9

📥 Commits

Reviewing files that changed from the base of the PR and between fc35290 and e4b78f5.

⛔ Files ignored due to path filters (65)
  • console/package.json is excluded by !console/**
  • console/src/cli/handlers/user-message.ts is excluded by !console/**
  • console/src/services/worker-service.ts is excluded by !console/**
  • console/src/services/worker/http/routes/AnnotationRoutes.ts is excluded by !console/**
  • console/src/services/worker/http/routes/LicenseRoutes.ts is excluded by !console/**
  • console/src/services/worker/http/routes/PlanRoutes.ts is excluded by !console/**
  • console/src/services/worker/http/routes/ViewerRoutes.ts is excluded by !console/**
  • console/src/services/worker/http/routes/utils/annotationStore.ts is excluded by !console/**
  • console/src/services/worker/http/routes/utils/planFileReader.ts is excluded by !console/**
  • console/src/ui/viewer/components/LicenseBadge.tsx is excluded by !console/**
  • console/src/ui/viewer/hooks/useLicense.ts is excluded by !console/**
  • console/src/ui/viewer/layouts/Sidebar/SidebarNav.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Changes/DiffPanel.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Changes/index.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Changes/review/CodeReviewPanel.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Changes/review/CodeReviewWidget.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Changes/review/types.ts is excluded by !console/**
  • console/src/ui/viewer/views/Changes/review/useCodeReview.ts is excluded by !console/**
  • console/src/ui/viewer/views/Spec/SpecHeaderCard.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/SpecSection.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/AnnotationPanel.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/AnnotationToolbar.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/BlockRenderer.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/PlanAnnotator.tsx is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/index.ts is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/parser.ts is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/types.ts is excluded by !console/**
  • console/src/ui/viewer/views/Spec/annotation/useAnnotation.ts is excluded by !console/**
  • console/src/ui/viewer/views/Spec/index.tsx is excluded by !console/**
  • console/tests/annotation/code-review.test.ts is excluded by !console/**
  • console/tests/annotation/parser.test.ts is excluded by !console/**
  • console/tests/annotation/use-annotation.test.ts is excluded by !console/**
  • console/tests/server/license-routes.test.ts is excluded by !console/**
  • console/tests/ui/ChangesNavigation.test.ts is excluded by !console/**
  • console/tests/ui/license-badge.test.ts is excluded by !console/**
  • console/tests/worker/annotation-routes.test.ts is excluded by !console/**
  • console/tests/worker/license-routes.test.ts is excluded by !console/**
  • console/tests/worker/utils/isValidPlanPath.test.ts is excluded by !console/**
  • launcher/cli.py is excluded by !launcher/**
  • launcher/tests/unit/test_cli.py is excluded by !launcher/**
  • launcher/tests/unit/test_worktree.py is excluded by !launcher/**
  • launcher/tests/unit/test_wrapper.py is excluded by !launcher/**
  • launcher/worktree.py is excluded by !launcher/**
  • launcher/wrapper.py is excluded by !launcher/**
  • pilot/commands/create-skill.md is excluded by !pilot/**
  • pilot/commands/spec-bugfix-plan.md is excluded by !pilot/**
  • pilot/commands/spec-bugfix-verify.md is excluded by !pilot/**
  • pilot/commands/spec-implement.md is excluded by !pilot/**
  • pilot/commands/spec-plan.md is excluded by !pilot/**
  • pilot/commands/spec-verify.md is excluded by !pilot/**
  • pilot/commands/spec.md is excluded by !pilot/**
  • pilot/rules/agent-browser.md is excluded by !pilot/**
  • pilot/rules/development-practices.md is excluded by !pilot/**
  • pilot/rules/playwright-cli.md is excluded by !pilot/**
  • pilot/rules/standards-backend.md is excluded by !pilot/**
  • pilot/rules/standards-frontend.md is excluded by !pilot/**
  • pilot/rules/task-and-workflow.md is excluded by !pilot/**
  • pilot/rules/testing.md is excluded by !pilot/**
  • pilot/rules/verification.md is excluded by !pilot/**
  • pilot/scripts/mcp-server.cjs is excluded by !pilot/**
  • pilot/scripts/worker-service.cjs is excluded by !pilot/**
  • pilot/settings.json is excluded by !pilot/**
  • pilot/ui/PlanAnnotator.js is excluded by !pilot/**
  • pilot/ui/viewer-bundle.js is excluded by !pilot/**
  • pilot/ui/viewer.css is excluded by !pilot/**
📒 Files selected for processing (22)
  • .gitignore
  • LICENSE
  • README.md
  • docs/docusaurus/.gitignore
  • docs/docusaurus/docs/features/cli.md
  • docs/docusaurus/docs/features/console.md
  • docs/docusaurus/docs/features/extensions.md
  • docs/docusaurus/docs/features/open-source-tools.md
  • docs/docusaurus/docs/features/rules.md
  • docs/docusaurus/docs/getting-started/installation.md
  • docs/docusaurus/docs/workflows/spec.md
  • docs/site/src/components/ConsoleSection.tsx
  • docs/site/src/components/FAQSection.tsx
  • docs/site/src/components/PricingSection.tsx
  • docs/site/src/components/WhatsInside.tsx
  • installer/steps/dependencies.py
  • installer/steps/finalize.py
  • installer/steps/shell_config.py
  • installer/tests/unit/steps/test_dependencies.py
  • installer/tests/unit/steps/test_dependencies_playwright.py
  • installer/tests/unit/steps/test_shell_config.py
  • uninstall.sh

Walkthrough

This PR introduces agent-browser as a replacement for Playwright CLI, establishes a new Enterprise license tier with source code access, documents new Console annotation and code review workflows, adds headless mode support to the CLI, and enhances installer robustness for edge cases in shell configuration handling.

Changes

Cohort / File(s) Summary
Browser Automation Tool Swap
.gitignore, docs/docusaurus/.gitignore, installer/steps/dependencies.py, installer/tests/unit/steps/test_dependencies.py, installer/tests/unit/steps/test_dependencies_playwright.py, uninstall.sh, docs/docusaurus/docs/features/open-source-tools.md, docs/docusaurus/docs/features/rules.md, docs/docusaurus/docs/getting-started/installation.md, docs/site/src/components/FAQSection.tsx, README.md
Replaces Playwright CLI with agent-browser throughout installer logic, tests, and documentation. Removes Playwright system dependency installation; adds subprocess-based agent-browser version checking and conditional upgrade/install flows per platform. Updates cache directory patterns and dependency licensing references.
Enterprise Tier & Licensing
LICENSE, docs/site/src/components/PricingSection.tsx, docs/docusaurus/docs/features/extensions.md, README.md
Adds Enterprise Source Code License (Section 2(e)) for organizations with 100\+ developer seats, grants full source access and derivative work rights for internal use, and replaces prior derivative work prohibition. Introduces new "Enterprise" pricing card with custom pricing, feature list, and apply form link. Updates tiering messaging to reflect free 7-day trial across all plans (no credit card required).
Console Annotation & Code Review Workflows
docs/docusaurus/docs/features/console.md, docs/site/src/components/ConsoleSection.tsx, docs/docusaurus/docs/workflows/spec.md
Documents new plan annotation mode in Specifications tab (free-text notes, visual markup, agent reading at approval) and code review mode in Changes tab (inline diff-line annotations, agent processing before verification). Updates spec workflow phases to require E2E test scenario documentation, run automated verification with remediation, and integrate user review checkpoints.
Headless Mode & CLI Enhancements
docs/docusaurus/docs/features/cli.md, docs/site/src/components/WhatsInside.tsx, README.md
Adds comprehensive headless mode documentation (pilot -p flag), usage examples, and Claude CLI flag compatibility notes. Introduces new "Headless Mode" grid card to marketing site with Cpu icon and description.
Installer Robustness
installer/steps/shell_config.py, installer/tests/unit/steps/test_shell_config.py, installer/steps/finalize.py
Improves shell config file parsing by using errors="replace" to handle non-UTF-8 bytes in alias detection/removal. Adds pre-success cleanup in finalize step to kill stale worker processes on port 41777.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

released

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title directly and specifically describes the main features added: plan annotations, code review mode, and E2E test scenarios in the spec workflow, which are confirmed as primary changes throughout the raw summary.
Docstring Coverage ✅ Passed Docstring coverage is 98.39% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

…workflow

Console — Plan Annotation UI (Spec tab):
- Block-based markdown rendering with native Selection API in Specifications tab
- Select any text in annotate mode → write free-text note → auto-saves to JSON immediately
- Single annotation type (free-text only — no DELETION/INSERTION/REPLACEMENT sub-types)
- Removed type toolbar and 'Add general comment' button
- View/Review toggle next to Specifications title; smart default (annotate for unapproved PENDING, view otherwise)
- AnnotationPanel, AnnotationToolbar, BlockRenderer, PlanAnnotator, useAnnotation, parser components

Console — Code Review Mode (Changes tab):
- View/Review toggle moved to Changes page header (previously buried in DiffPanel)
- Inline diff annotations: click '+' on any hunk line, write note, auto-saves to unified JSON
- Staged/unstaged counts in file list sidebar header
- CodeReviewPanel, CodeReviewWidget, useCodeReview components extracted

Console — Unified Annotation Storage:
- Single JSON per spec at docs/plans/.annotations/{plan-filename}.json
- Both planAnnotations and codeReviewAnnotations in one file
- Auto-save on every change (debounced) — no 'Send Feedback' button anywhere
- Removed feedback markdown file generation entirely
- AnnotationRoutes.ts and annotationStore.ts refactored to unified model
- Active spec resolution for Changes tab scopes code review annotations per spec

Console — Sidebar + Layout:
- Sidebar reorder: Dashboard → Changes → Specifications → Extensions → ...
- Changes and Specifications now adjacent for the plan→review→verify flow

Spec Commands — E2E Test Scenarios:
- spec-plan Step 1.5.2: write structured TS-NNN scenarios during planning for UI features
  (step table, priority, preconditions, mapped tasks — agent-browser executable)
- Task structure gains 'Mapped Scenarios' field; plan template gains '## E2E Test Scenarios' section
- spec-verify Step 3.9 refactored: executes plan's TS-NNN scenarios with TaskCreate tracking,
  fix attempts (max 2 → known issue escalation), results written to '## E2E Results' in plan file
- spec-bugfix-plan: Verification Scenario for UI-facing bugs (single TS-001 acceptance scenario)
- spec-bugfix-verify Step 3.5b: executes verification scenario, loops back on 2-attempt failure
- Console Spec view: E2E Test Scenarios, E2E Results, Verification Scenario added to DISPLAYED_SECTIONS with icons

Spec Commands — Annotation JSON Integration:
- spec-plan Step 1.7b and spec-bugfix-plan Step 1.4b read planAnnotations from unified JSON
- spec-verify Step 3.12b and spec-bugfix-verify Step 3.7b read codeReviewAnnotations from unified JSON
- All annotation feedback steps updated to DELETE via new /api/annotations endpoints

Launcher — Worktree Sync Auto-Stash:
- sync_worktree() now auto-stashes uncommitted changes on base branch before merging
- Stash is preserved (never dropped) if restore encounters conflicts
- spec-verify and spec-bugfix-verify: removed pre-sync working tree cleanliness check
- Fixes GitHub issue #119 ('Main branch has an unstashed file')

Rules + Commands — playwright-cli → agent-browser:
- pilot/rules/playwright-cli.md removed; pilot/rules/agent-browser.md added
- All spec commands, rules, and settings updated to reference agent-browser

Installer:
- finalize.py: kill stale Console worker (port 41777) on install so new files take effect immediately
- dependencies.py: agent-browser replaces playwright-cli in dependency checks
- shell_config.py: encoding improvements

Other:
- LICENSE: Enterprise Source Code License clause for 100+ seat organizations
- settings.json: autoDreamEnabled; removed CLAUDE_CODE_MAX_OUTPUT_TOKENS/EFFORT_LEVEL overrides; agent-browser tip
- pilot/ui/: viewer bundle rebuilt with annotation components (PlanAnnotator.js)
- README, Docusaurus, site: updated spec workflow descriptions, E2E scenarios, console features
- docs/docusaurus: remote-control.md removed, console.md expanded with annotation features
@claude
Copy link
Copy Markdown

claude bot commented Mar 27, 2026

Claude finished @maxritter's task in 1m 46s —— View job


Code Review

Incremental review — focusing on new changes in this push. Encrypted paths (console/**, launcher/**) skipped per configuration.


New Issues

1. install_agent_browser() silently ignores upgrade failureinstaller/steps/dependencies.py

if _is_agent_browser_ready():
    _run_bash_with_retry("agent-browser upgrade", timeout=120)
    return True  # always True, even if upgrade failed

The result of _run_bash_with_retry is discarded. If the upgrade fails (network error, permission issue), the function reports success. Since the tool is present, this may be intentional best-effort, but it's inconsistent with the rest of the function which returns False on install failures. At minimum, a log warning when the upgrade fails would help debugging. Fix this →


2. _kill_stale_worker() uses kill -9 without graceful shutdown attemptinstaller/steps/finalize.py

subprocess.run(["kill", "-9", pid.strip()], capture_output=True, timeout=5)

SIGKILL skips signal handlers, connection cleanup, and in-flight writes. If the Console worker is mid-write to its SQLite store, this could corrupt the database. The standard approach is SIGTERM first (graceful), then SIGKILL after a short timeout if the process hasn't exited. Also, lsof may not be available in all install environments (e.g., minimal Linux containers). Fix this →


3. Fragile sed-based JSON parsing in spec-verify.md

The previous python3 -c "import sys,json; print(json.load(sys.stdin)['project_root'])" was replaced with:

| sed -n 's/.*"project_root": *"\([^"]*\)".*/\1/p'

This regex breaks if the path contains spaces, backslashes, or if the JSON is minified differently. Python is already a guaranteed dependency in this project (it's checked in prerequisites). The original approach was more robust. Fix this →


Positive Changes

  • shell_config.py: Adding errors="replace" to read_text() is a solid robustness fix for shell configs with non-UTF-8 bytes — the new tests cover both detection and removal paths well.
  • Playwright → agent-browser migration: Clean replacement, the _is_agent_browser_ready() simplification (version check vs. filesystem cache inspection) is more portable and reliable.
  • E2E test scenarios in spec-plan.md: Structured ## E2E Test Scenarios format with explicit TS-NNN IDs that map through to spec-verify.md execution is a nice traceability improvement over ad-hoc verification.
  • Code review annotation gate: The Step 3.12b / Step 3.7b pattern (check JSON annotations before marking VERIFIED) creates a clean human-in-the-loop checkpoint without requiring a separate tool or button.
  • Performance audit step added to both spec-verify.md and spec-bugfix-verify.md: Making it explicit ("static code review — no running program needed") sets the right expectation.

Still Open

No prior review comments on this PR.

@maxritter
Copy link
Copy Markdown
Owner Author

Recreating PR with clean history.

@maxritter maxritter closed this Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant