feat(architect): generate → run → judge → refine loop that ships Proven templates#269
Closed
gizmax wants to merge 3 commits into
Closed
feat(architect): generate → run → judge → refine loop that ships Proven templates#269gizmax wants to merge 3 commits into
gizmax wants to merge 3 commits into
Conversation
… proof bundles engine/architect.py drives the loop: generate a workflow from a NL description via the existing generator, execute it live with cassette recording on (budget- and time-capped, admin_trusted=False), hard-check the run and LLM-judge the output against the description, and feed failures back through the generator's refine path. On success the workflow + recorded cassette are packed into a .sctpl bundle, verified with the same verify_bundle that gates template installs, and installed into the community templates dir. Config: architect_max_iterations (3), architect_budget_usd (1.0), architect_score_threshold (0.7) with clamping validators. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /api/architect starts an admin-gated async job (in-memory job
store, same pattern as batch runs); GET /api/architect/{job_id} returns
status, the per-iteration progress log, and the final result with
bundle path. CLI: sandcastle architect "<description>" [--input k=v]
[--budget] [--max-iterations] [--threshold] runs the loop synchronously,
prints iteration progress, exits 0 with the bundle path on success and
2 when the loop ends unproven.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… section Tests mock the LLM surfaces (generator + judge) but run the real executor against a mocked provider runtime, so the recorded cassette and packed bundle are genuine - the success test proves the bundle with verify_bundle. Covers: first-iteration success, refine-then-succeed, validation-error feedback, budget exceeded aborts cleanly, max iterations returns best-effort proven=false, hard-check failures skip the judge, test-input derivation, config defaults, API job lifecycle (start/poll/404/failure), CLI exit codes. README: 'The Architect' subsection under Verified Templates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Shipped in v0.40.0 (release integration merged to main). Closing — superseded by the 0.40.0 release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades the one-shot
/generateendpoint into The Architect: an autonomous loop that doesn't stop until the generated workflow actually works — and ships it with a recorded cassette proof.engine/architect.py: generate (existinggenerator.py, reusing itsrefine_from/refine_instruction) → derive test inputs (schema defaults → LLM → placeholders) → execute for real with cassette recording, budget- and time-capped,admin_trusted=False→ hard checks + LLM judge vs the original description → refine with judge feedback, up toarchitect_max_iterations(3)bundle.pyand verified with the sameverify_bundlethat gates installs —proven=trueonly if the proof actually verifies; installed liketemplate installPOST /api/architect(async job) +GET /api/architect/{job_id}(per-iteration log); CLI:sandcastle architect "<description>" [--budget 1.0]standardsteps; generatedcodesteps can never execute (admin_trusted=False); budget tracks live-run spend with per-run capsTests: 18 new + 485-test regression subset green (incl. template bundle suite); ruff clean.
Stacked on #264 (template-hub) — merge that first.