docs: improve API reference to match actual codebase - #117
Conversation
Rewrite docs/api/index.md to document the real public API surface: - CLI commands (init, config, pipeline, start, doctor, estimate-cost, etc.) - Core Python API (EvolutionPipeline, EvolutionConfig) - Safety layer (SafetyIntegration, CheckpointManager, EditScopeValidator, etc.) - Phase 3 services (ContinuousEvolutionService, TrainingManager) - Monitoring dashboard quick reference with auth/CORS info The previous version documented fabricated classes (EVOSEAL.evolve(), OpenAIModel, AnthropicModel) that don't exist in the codebase. Also fix stale CORS and auth info in docs/API_REFERENCE.md: - Auth: document optional bearer token (added in PR #109) - CORS: origins now default to host:port, not wildcard * Addresses TODO.md item: Improve API reference
📝 WalkthroughWalkthroughThe documentation now describes the current Typer CLI, evolution and safety APIs, continuous services, dashboard interfaces, authentication, CORS, and security guidance. The API reference task is marked complete. ChangesAPI Documentation
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/API_REFERENCE.md`:
- Line 322: Update the Dashboard default-host documentation to reflect that
MonitoringDashboard.__init__ uses "localhost" rather than asserting the literal
IPv4 address 127.0.0.1, unless the binding implementation explicitly guarantees
IPv4 loopback resolution.
- Line 331: Update the Origins documentation to state that when the dashboard
binds to wildcard hosts 0.0.0.0 or ::, the default CORS origin is
http://localhost:<port>; otherwise it uses the dashboard host and port, never
the wildcard bind address itself.
- Line 316: Update the WebSocket authentication entry in API_REFERENCE.md to
document both the token query parameter and the Sec-WebSocket-Protocol
bearer.<token> handshake mechanism supported by monitoring_dashboard.py. Present
them as supported alternatives without implying that ?token=<value> is the sole
or preferred transport.
In `@docs/api/index.md`:
- Around line 74-81: Update the EvolutionPipeline documentation examples in
docs/api/index.md to use valid Python dictionaries for dgm_config,
openevolve_config, seal_config, and test_config, replacing each `{...}`
placeholder—including the example near line 178—with `{}` or representative
key-value data.
- Around line 84-89: Update the asynchronous examples around
pipeline.run_evolution_cycle and pipeline.run_evolution_cycle_with_safety, plus
the corresponding examples near the later documented lines, so they are valid in
a normal Python module by placing them in an async entry function and showing
its asyncio.run invocation; keep the existing example behavior unchanged.
- Around line 204-228: Update the Monitoring Dashboard section’s port
description to match the MonitoringDashboard constructor default of 8081, or
explicitly document the launcher/environment override that changes it to 9613;
keep the endpoint and authentication documentation unchanged.
- Around line 167-201: Correct the Phase 3 API documentation for
ContinuousEvolutionService and TrainingManager: describe start() as entering the
awaited _run_service_loop while shutdown() signals graceful termination without
blocking it, and document the dict[str, Any] return types for all listed
TrainingManager methods with no required parameters. Add the full constructor
signatures, removing the unsupported config argument from TrainingManager
examples and showing the required arguments for both services.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3ab85622-2bf4-41a4-826c-d4644715c2ea
📒 Files selected for processing (3)
TODO.mddocs/API_REFERENCE.mddocs/api/index.md
| present a valid token: | ||
|
|
||
| - **HTTP**: `Authorization: Bearer <token>` header | ||
| - **WebSocket**: `?token=<value>` query parameter |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
rg -n -C 12 'def _auth_middleware|Authorization|bearer\.|query.*token|WebSocket' evoseal/services/monitoring_dashboard.pyRepository: SHA888/EVOSEAL
Length of output: 14684
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== API_REFERENCE WebSocket lines =="
sed -n '300,325p' docs/API_REFERENCE.md
echo
echo "== auth/websocket tests mentioning dashboard websocket =="
git ls-files | rg '(test|tests|spec)' | xargs -r rg -n -i 'monitoring_dashboard|MonitoringDashboard|websocket|Bearer|bearer\.|Sec-WebSocket-Protocol|auth_token|query.*token' || true
echo
echo "== static verifier for supported WebSocket auth branches and docs token phrasing =="
python3 - <<'PY'
from pathlib import Path
import re
service = Path("evoseal/services/monitoring_dashboard.py").read_text()
md = Path("docs/API_REFERENCE.md").read_text()
auth = service[service.find("async def _auth_middleware"):service.find(" `@staticmethod`", service.find("async def _auth_middleware"))]
ws_handler = service[service.find("async def websocket_handler"):service.find(" async def _get_current_metrics", service.find("async def websocket_handler"))]
dashboard_js = service[service.find(" async def dashboard_page"):service.find(" return \"\"\"", service.find(" async def __call__", service.find(" async def dashboard_page")))]
checks = {
"middleware_checks Authorization bearer": "Authorization" in auth and "Bearer {self.auth_token}" in auth,
"middleware checks Sec-WebSocket-Protocol bearer": "Sec-WebSocket-Protocol" in auth and 'proto.startswith("bearer.")' in auth,
"middleware checks query token": "request.query.get(\"token\", \"\")" in auth,
"route /ws added": 'self.app.router.add_get("/ws", self.websocket_handler)' in service,
"handler selects Sec-WebSocket-Protocol bearer": "Sec-WebSocket-Protocol" in ws_handler and 'proto.startswith("bearer.")' in ws_handler,
"js uses bearer subprotocol": re.search(r"WebSocket\(wsUrl,\s*authToken\s*\?\s*\['bearer\.\'\s*\+\s*authToken\s*\]\s*:\s*undefined\)", service) is not None,
"js comment says header not URL/logs": "Sec-WebSocket-Protocol" in dashboard_js and "access logs" in dashboard_js,
"docs line says ?token=<value>": "?token=<value>" in md,
}
for name, ok in checks.items():
print(f"{name}: {ok}")
# Extract exact documentation line around dashboard API token.
for i, line in enumerate(md.splitlines(), 1):
if "?token=" in line or "WebSocket" in line and "token" in line:
pos = line.find("WebSocket")
print(f"doc_hit_{i}: {line[pos:pos+80] if pos!=-1 else line}")
PYRepository: SHA888/EVOSEAL
Length of output: 8553
Document the supported WebSocket auth transport correctly.
doc/API_REFERENCE.md:316 currently says only ?token=<value>, but evoseal/services/monitoring_dashboard.py supports Sec-WebSocket-Protocol: bearer.<token> as the header-based handshake path and the embedded dashboard JS uses that path. Update the API reference to list the supported WebSocket mechanisms and avoid making ?token=<value> look like the only or preferred option.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/API_REFERENCE.md` at line 316, Update the WebSocket authentication entry
in API_REFERENCE.md to document both the token query parameter and the
Sec-WebSocket-Protocol bearer.<token> handshake mechanism supported by
monitoring_dashboard.py. Present them as supported alternatives without implying
that ?token=<value> is the sole or preferred transport.
| ### Security Considerations | ||
| - Dashboard binds only to localhost (127.0.0.1) | ||
| - No external network access | ||
| - Dashboard defaults to localhost (127.0.0.1) binding |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Use the actual default host value.
MonitoringDashboard.__init__ defaults host to "localhost", not the literal "127.0.0.1". Replace the parenthetical unless the binding implementation guarantees IPv4 loopback resolution. This avoids misleading operators on systems where localhost resolves to ::1.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/API_REFERENCE.md` at line 322, Update the Dashboard default-host
documentation to reflect that MonitoringDashboard.__init__ uses "localhost"
rather than asserting the literal IPv4 address 127.0.0.1, unless the binding
implementation explicitly guarantees IPv4 loopback resolution.
| - **Credentials**: Allowed | ||
| Cross-Origin Resource Sharing (CORS) is configured as follows: | ||
|
|
||
| - **Origins**: Defaults to the dashboard's own `host:port` (not `*`) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Document the wildcard-bind CORS exception.
evoseal/services/monitoring_dashboard.py:54-138 sets the default origin to http://localhost:<port> when host is 0.0.0.0 or ::. It does not use the wildcard host value as the default origin.
Proposed wording
-- **Origins**: Defaults to the dashboard's own `host:port` (not `*`)
+- **Origins**: Defaults to the dashboard origin; for `0.0.0.0` or `::`, defaults to `http://localhost:<port>` (not `*`)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - **Origins**: Defaults to the dashboard's own `host:port` (not `*`) | |
| - **Origins**: Defaults to the dashboard origin; for `0.0.0.0` or `::`, defaults to `http://localhost:<port>` (not `*`) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/API_REFERENCE.md` at line 331, Update the Origins documentation to state
that when the dashboard binds to wildcard hosts 0.0.0.0 or ::, the default CORS
origin is http://localhost:<port>; otherwise it uses the dashboard host and
port, never the wildcard bind address itself.
| pipeline = EvolutionPipeline( | ||
| EvolutionConfig( | ||
| dgm_config={...}, | ||
| openevolve_config={...}, | ||
| seal_config={...}, | ||
| test_config={...}, | ||
| max_iterations=1000, | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python - <<'PY'
value = {...}
assert isinstance(value, set)
assert value == {Ellipsis}
print("verified: {...} is a set containing Ellipsis")
PYRepository: SHA888/EVOSEAL
Length of output: 197
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf 'File outline/size:\n'
wc -l docs/api/index.md
printf '\nRelevant Python examples (lines 50-90 and 170-185):\n'
sed -n '50,90p;170,185p' docs/api/index.md | nl -ba -v50
printf '\nOccurrences of {...} in docs/api/index.md:\n'
grep -n '{\.\.\.}' docs/api/index.md || trueRepository: SHA888/EVOSEAL
Length of output: 286
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf 'Relevant Python examples with line numbers:\n'
sed -n '50,90p;170,185p' docs/api/index.md | awk '{printf "%03d\t%s\n", NR+50, $0}'
printf '\nOccurrences of {...} in docs/api/index.md:\n'
grep -nE '\{\.\.\.\}' docs/api/index.md || true
printf '\nOccurrences of await in docs/api/index.md:\n'
grep -nE '\bawait\b' docs/api/index.md || trueRepository: SHA888/EVOSEAL
Length of output: 2486
Use real dictionaries in the Python docs examples.
{...} is a set containing Ellipsis, not a dictionary. Replace the placeholders at docs/api/index.md:75-81 and docs/api/index.md:178 with {} or representative key-value config data.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/api/index.md` around lines 74 - 81, Update the EvolutionPipeline
documentation examples in docs/api/index.md to use valid Python dictionaries for
dgm_config, openevolve_config, seal_config, and test_config, replacing each
`{...}` placeholder—including the example near line 178—with `{}` or
representative key-value data.
| # Run one or more evolution iterations | ||
| results = await pipeline.run_evolution_cycle(iterations=1) | ||
|
|
||
| # Run with safety gates (checkpoint + rollback on regression) | ||
| results = await pipeline.run_evolution_cycle_with_safety(iterations=1) | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python - <<'PY'
source = "results = await pipeline.run_evolution_cycle(iterations=1)\n"
try:
compile(source, "<documentation>", "exec")
except SyntaxError:
print("verified: top-level await is rejected in a normal module")
else:
raise SystemExit("unexpected: top-level await compiled")
PYRepository: SHA888/EVOSEAL
Length of output: 209
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the referenced documentation sections.
fd -a 'index\.md$' . | sed 's#^\./##'
echo '--- docs/api/index.md outline/line count ---'
wc -l docs/api/index.md || true
echo '--- lines 70-95 and 168-188 ---'
sed -n '70,95p;168,188p' docs/api/index.mdRepository: SHA888/EVOSEAL
Length of output: 1760
Make the asynchronous examples valid in a normal Python module.
The examples at lines 84-89 use top-level await, which is only valid in async-capable REPLs, or they are valid only if they are inside async def main() and launched with asyncio.run(main()). Lines 179-180 have the same issue. Wrap these snippets in an async entry function, or explicitly state they must run in async mode.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/api/index.md` around lines 84 - 89, Update the asynchronous examples
around pipeline.run_evolution_cycle and
pipeline.run_evolution_cycle_with_safety, plus the corresponding examples near
the later documented lines, so they are valid in a normal Python module by
placing them in an async entry function and showing its asyncio.run invocation;
keep the existing example behavior unchanged.
| ## Phase 3 — Continuous Evolution | ||
|
|
||
| ### `ContinuousEvolutionService` | ||
|
|
||
| Long-running daemon that drives the autonomous evolution → training → deployment loop. | ||
|
|
||
| **Module:** `evoseal.services.continuous_evolution_service` | ||
|
|
||
| ```python | ||
| from evoseal.services.continuous_evolution_service import ContinuousEvolutionService | ||
|
|
||
| service = ContinuousEvolutionService(config={...}) | ||
| await service.start() # enters the service loop | ||
| await service.shutdown() # graceful stop | ||
| ``` | ||
|
|
||
| | Method | Description | | ||
| |---|---| | ||
| | `start()` | Start the daemon (signal handlers, service loop) | | ||
| | `shutdown()` | Graceful shutdown | | ||
| | `generate_service_report()` | Return a status/evolution/training report dict | | ||
|
|
||
| ### `TrainingManager` | ||
|
|
||
| Manages fine-tuning readiness checks, data preparation, and training cycles. | ||
|
|
||
| **Module:** `evoseal.fine_tuning.training_manager` | ||
|
|
||
| | Method | Description | | ||
| |---|---| | ||
| | `check_training_readiness()` | Check if enough data exists and no training is in progress | | ||
| | `prepare_training_data()` | Build training dataset from evolution results | | ||
| | `run_training_cycle()` | Execute one full training cycle | | ||
| | `get_training_status()` | Return current training status | | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
fd --type f --extension py -0 |
xargs -0 rg -n -C 4 \
'class ContinuousEvolutionService\b|class TrainingManager\b|(?:async )?def (start|shutdown|generate_service_report|check_training_readiness|prepare_training_data|run_training_cycle|get_training_status)\b' || trueRepository: SHA888/EVOSEAL
Length of output: 15448
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
for file in \
ev-seal/services/continuous_evolution_service.py \
evoseal/services/continuous_evolution_service.py \
evoseal/fine_tuning/training_manager.py
do
if [ -f "$file" ]; then
echo "===== $file ====="
wc -l "$file"
fi
done
echo "===== continuous_evolution_service outline ====="
ast-grep outline evoseal/services/continuous_evolution_service.py --view expanded || true
echo "===== training_manager outline ====="
ast-grep outline evoseal/fine_tuning/training_manager.py --view expanded || true
echo "===== relevant continuous_evolution_service slices ====="
sed -n '1,220p' evoseal/services/continuous_evolution_service.py | cat -n
sed -n '380,480p' evoseal/services/continuous_evolution_service.py | cat -n
echo "===== relevant training_manager slices ====="
sed -n '1,340p' evoseal/fine_tuning/training_manager.py | cat -nRepository: SHA888/EVOSEAL
Length of output: 31753
Correct the API contract in Phase 3 docs.
ContinuousEvolutionService.start() enters the main service loop by await self._run_service_loop(), and shutdown() does not block it. TrainingManager methods return dict[str, Any] and take no required parameters, but the constructor does not accept config. Include the full constructor signatures where examples call these services.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/api/index.md` around lines 167 - 201, Correct the Phase 3 API
documentation for ContinuousEvolutionService and TrainingManager: describe
start() as entering the awaited _run_service_loop while shutdown() signals
graceful termination without blocking it, and document the dict[str, Any] return
types for all listed TrainingManager methods with no required parameters. Add
the full constructor signatures, removing the unsupported config argument from
TrainingManager examples and showing the required arguments for both services.
| ## Monitoring Dashboard | ||
|
|
||
| The Phase 3 monitoring dashboard serves a REST API and WebSocket interface on port 9613 | ||
| (default). Full endpoint documentation is in [API_REFERENCE.md](../API_REFERENCE.md). | ||
|
|
||
| ### Quick Reference | ||
|
|
||
| | Endpoint | Method | Description | | ||
| |---|---|---| | ||
| | `/` | GET | Dashboard HTML page | | ||
| | `/api/status` | GET | Service status + basic metrics | | ||
| | `/api/metrics` | GET | Comprehensive system metrics | | ||
| | `/api/report` | GET | Detailed evolution report | | ||
| | `/ws` | WS | Real-time metrics updates (every 30s) | | ||
|
|
||
| ### Authentication | ||
|
|
||
| When the dashboard is constructed with `auth_token`, all `/api/*` and `/ws` requests must | ||
| include `Authorization: Bearer <token>`. WebSocket clients may pass `?token=<value>` as a | ||
| query parameter instead. | ||
|
|
||
| ### CORS | ||
|
|
||
| Allowed origins default to the dashboard's own `host:port`. Binding to `0.0.0.0` logs a | ||
| security warning. Wildcard `*` origins disable `allow_credentials` per the CORS spec. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 4 \
'auth_token|Authorization|Bearer|token=|CORSMiddleware|allow_origins|allow_credentials|9613|/api/status|/api/metrics|/api/report|/ws' \
. || trueRepository: SHA888/EVOSEAL
Length of output: 50373
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf 'Files:\n'
fd -a 'monitoring_dashboard.py|run_phase3_continuous_evolution.py' . | sed 's#^\./##'
printf '\nMonitoring dashboard outline:\n'
ast-grep outline evoseal/services/monitoring_dashboard.py --view compact || true
printf '\nRelevant monitoring dashboard lines:\n'
nl -ba evoseal/services/monitoring_dashboard.py | sed -n '1,260p'Repository: SHA888/EVOSEAL
Length of output: 624
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo 'Run phase3 argument setup:'
awk '/def parse_args\(|--host|--port|--auth-token|auth_token|cors|host:port/ { for(i=NR-2;i<=NR+20;i++) print i":" f[i] }' f=evoseal/services/monitoring_dashboard.py scripts/lib/evolution/run_phase3_continuous_evolution.py 2>/dev/null | head -n 220 || true
echo
echo 'monitoring_dashboard.py sections of interest:'
python3 - <<'PY'
from pathlib import Path
p=Path('evoseal/services/monitoring_dashboard.py')
lines=p.read_text().splitlines()
for start,end in [(1,140),(140,260),(260,420),(420,560),(560,740),(740,880),(880,1060),(1060,1160)]:
print(f'\n--- {min(start,len(lines))}-{min(end,len(lines))} ---')
for i,line in enumerate(lines[start:end], start+1):
print(f'{i:4}: {line}')
PYRepository: SHA888/EVOSEAL
Length of output: 36312
Update the dashboard default port in deployment docs.
MonitoringDashboard defaults to port=8081, but docs/api/index.md:205-206 says the dashboard serves on port 9613. Update this section to match the constructor default, or document the external launcher/env override that provides 9613.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/api/index.md` around lines 204 - 228, Update the Monitoring Dashboard
section’s port description to match the MonitoringDashboard constructor default
of 8081, or explicitly document the launcher/environment override that changes
it to 9613; keep the endpoint and authentication documentation unchanged.
What
Rewrites
docs/api/index.mdto document the actual public API surface of EVOSEAL, replacing fabricated class stubs (EVOSEAL.evolve(),OpenAIModel,AnthropicModel) that don't exist in the codebase.Also fixes stale CORS and authentication info in
docs/API_REFERENCE.mdto reflect PR #109.Why
The previous
docs/api/index.mdwas misleading — it documented a made-upEVOSEALclass with anevolve()method and model wrapper classes that were never implemented. New contributors reading the API reference would get a completely wrong picture of the codebase.Changes
docs/api/index.md(full rewrite):init,config,pipeline,start,doctor,estimate-cost, subsystem commands)EvolutionPipeline,EvolutionConfigwith real method signaturesSafetyIntegration,CheckpointManager,EditScopeValidator,RegressionDetector,TestRunnerContinuousEvolutionService,TrainingManagerdocs/API_REFERENCE.md(targeted fixes):auth_tokenparameter)host:port, not*; wildcard disables credentials per specTODO.md: checked off "Improve API reference"Addresses: TODO.md item "Improve API reference" (line 285)
Summary by CodeRabbit