docs: add 'How It Actually Works' tutorial - #116
Conversation
Walk through a single evolution cycle step by step, covering: - Pipeline initialization and safety setup - The 7-stage iteration loop (analyze → generate → adapt → evaluate → validate) - Safety wraparound with checkpoints and rollback - Resilience and error recovery - Continuous daemon mode and the bidirectional loop Closes TODO.md item: 'Add a How It Actually Works tutorial'
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. 📝 WalkthroughWalkthroughThe PR adds a tutorial for EVOSEAL’s runtime evolution flow, links it from the documentation index, and updates TODO tracking to mark the tutorial complete. ChangesRuntime documentation
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/tutorials/how_it_works.md`:
- Around line 60-67: Update every fenced code block in
docs/tutorials/how_it_works.md, including the blocks around
EvolutionPipeline.__init__ and the referenced sections, with an explicit
language identifier. Use text for diagram blocks and python for the daemon loop,
preserving their existing contents.
- Around line 69-71: The tutorial presents contradictory safety guidance between
the safety-layer explanation and the run_evolution_cycle_with_safety section.
Reconcile the documentation around run_evolution_cycle() and
run_evolution_cycle_with_safety() by clearly stating which entry point applies
checkpoints and rollback, whether SafetyIntegration is mandatory, and which
function callers must use; update both affected sections consistently.
- Around line 188-201: Clarify the `_validate_improvement` documentation to
describe the fewer-than-two-metrics path as baseline establishment, explicitly
noting that the first iteration bypasses
`ImprovementValidator.validate_improvement`; only change the implementation if
the intended contract requires first candidates to pass that gate.
- Around line 122-124: The tutorial’s description of the evolutionary search
overstates the current implementations of _generate_improvements() and
_evaluate_version(). Update the relevant section to describe these methods as
placeholders returning empty/default results, or otherwise accurately explain
the adapter-driven workflow only where it is implemented.
In `@TODO.md`:
- Around line 282-284: Do not mark the “How It Actually Works” tutorial complete
in TODO.md until docs/tutorials/how_it_works.md includes a real CLI dry-run log
transcript alongside its walkthrough and payload examples. Add the transcript,
or revise the TODO requirement to match the tutorial’s actual content before
updating any completion counts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5695a0f7-889a-4a9a-bef0-9a778085a7a0
📒 Files selected for processing (3)
TODO.mddocs/index.mddocs/tutorials/how_it_works.md
| This is the gate. The `ImprovementValidator` compares the current iteration's | ||
| metrics against the previous iteration's metrics. If the new version | ||
| regresses on any critical dimension, the improvement is rejected and the | ||
| cycle stops (`should_continue=False`). | ||
|
|
||
| ```python | ||
| # From _validate_improvement (simplified): | ||
| metrics = self.metrics_tracker.get_metrics_history(test_type) | ||
|
|
||
| if len(metrics) < 2: | ||
| return True # first iteration, nothing to compare against | ||
|
|
||
| result = self.validator.validate_improvement(baseline_id, comparison_id, test_type) | ||
| return result.passed |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Clarify first-iteration validation.
Lines 197-198 return True when fewer than two metrics exist. The first iteration therefore bypasses ImprovementValidator.validate_improvement at Line 200. Describe this as baseline establishment, or change the implementation if the first candidate must pass the validation gate.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/tutorials/how_it_works.md` around lines 188 - 201, Clarify the
`_validate_improvement` documentation to describe the fewer-than-two-metrics
path as baseline establishment, explicitly noting that the first iteration
bypasses `ImprovementValidator.validate_improvement`; only change the
implementation if the intended contract requires first candidates to pass that
gate.
| - [x] **Add a "How It Actually Works" tutorial** | ||
| - Walk through a single evolution cycle step by step with real logs | ||
| - Lower the barrier for new contributors |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not mark the tutorial complete before meeting its stated criteria.
The TODO requires a walkthrough “with real logs.” docs/tutorials/how_it_works.md contains snippets and example payloads, but no real CLI log transcript. Add a dry-run transcript or revise the TODO wording before updating the completion counts.
Also applies to: 311-312
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@TODO.md` around lines 282 - 284, Do not mark the “How It Actually Works”
tutorial complete in TODO.md until docs/tutorials/how_it_works.md includes a
real CLI dry-run log transcript alongside its walkthrough and payload examples.
Add the transcript, or revise the TODO requirement to match the tutorial’s
actual content before updating any completion counts.
…torial - Step 2 comment: relabel from _run_single_iteration to run_evolution_cycle (the snippet is the loop body of run_evolution_cycle, not the inner method) - __init__ line number: ~140 → ~95 - SandboxedTestRunner line number: ~630 → ~595
- Rewrite Steps 4-6 to clarify that pipeline methods are stubs and real logic lives in integration adapters (Controller, SEAL, SandboxedTestRunner), matching the 'What's Still TODO' section framing - Remove all line number references (line ~N) from Key file annotations to prevent drift from misleading new contributors - Add 'Illustrative' disclaimers to all pseudo-code blocks that are not exact source excerpts - Update 'What's Still TODO' section to cross-reference the adapter descriptions in Steps 4-6 instead of contradicting them - Further Reading links all verified to exist — no changes needed
|
Addressed all four review points in 01b4872: 1. Steps vs. TODO contradiction (Steps 4-6): Rewrote Steps 4, 5, and 6 to match the adapter-centric framing. Each now explicitly states the pipeline method is a stub/passthrough and describes where the real logic lives (Controller.run_generation for generation, SEAL adapter for adaptation, SandboxedTestRunner for evaluation). The 'What's Still TODO' section now cross-references these same adapter descriptions instead of contradicting them. 2. Line number references: Removed all annotations from Key file references. The function/class names remain. Line numbers were accurate at time of writing but will drift. 3. Further Reading links: Verified all four paths exist on disk — no changes needed:
4. Code snippet disclaimers: Added explicit / comments to all pseudo-code blocks (Steps 2, 4, 7, 8, 10) that aren't exact source excerpts. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/tutorials/how_it_works.md`:
- Around line 329-336: Update the documentation around the pipeline methods to
distinguish their extension points: generation and adaptation delegate to the
DGM, OpenEvolve, and SEAL integration adapters, while evaluation is implemented
by SandboxedTestRunner. Avoid stating that all real logic resides only in the
integration adapters.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 14baa77e-ee30-431e-8a9c-a46691c57de4
📒 Files selected for processing (1)
docs/tutorials/how_it_works.md
Address PR #116 review feedback: - Clarify that Controller.run_generation is a separate code path, not called by _generate_improvements() or on the pipeline's execution path. Remove misleading 'independent inner loop' framing. - Resolve safety mandatory/optional contradiction: Step 1's SafetyIntegration is always initialized; Step 8's checkpoint/rollback wrapping is a separate optional variant (run_evolution_cycle_with_safety). - Add language identifiers to all fenced code blocks (text/python). - Distinguish extension points in What's Still TODO: generation and adaptation use integration adapters; evaluation uses SandboxedTestRunner. - Update Key file reference for Step 4 to reflect both pipeline stub and Controller as separate code path.
Run ruff format to remove trailing whitespace on line 281 that was causing the 'Lint and Type Check' CI job to fail.
…ks tutorial - Fix STAGE_STARTED/STAGE_COMPLETED → PIPELINE_STAGE_STARTED/PIPELINE_STAGE_COMPLETED (verified against events.py lines 117-118) - Standardize all illustrative code block comments to 'Illustrative — see ...' format
…orial # Conflicts: # TODO.md
1 participant
What
Add a step-by-step tutorial walking through a single EVOSEAL evolution cycle, from initialization through validation.
Why
The TODO item 'Add a How It Actually Works tutorial' calls for lowering the barrier for new contributors by showing the runtime flow with real code references instead of requiring them to read the source cold.
What's covered
Changes
Verification
Closes TODO.md item: 'Add a How It Actually Works tutorial'
Summary by CodeRabbit