You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of #4189. Builds on the config gate in #4190 (no hard code dependency — this is a standalone pure/async module testable against fixture diffs — but ships gated by the same flag).
Context
gittensory already has an established pattern for LLM-generated advisory content that reuses the shared model-calling plumbing in src/services/ai-review.ts:
callAiProvider (ai-review.ts:1178) is the single call site every AI-generated artifact goes through.
BEST_REVIEW_MODELS / RELIABLE_FALLBACK_MODELS (ai-review.ts:46-56) are the default model tiers; a repo with BYOK configured uses "the maintainer's own frontier model (Anthropic/OpenAI)" instead — see the exact pattern in src/services/ai-slop.ts:219,252 (model: input.providerKey ? \byok:${input.providerKey.provider}` : WORKERS_SLOP_MODELS.join("+")`), which is how BYOK-vs-default model selection and cost-event labeling already work for the slop advisory writer.
estimateNeurons (ai-review.ts:493) and BYOK_MODEL_PRICING_USD_PER_MTOK (ai-review.ts:1097) are the existing cost-accounting primitives.
buildTestEvidencePromptSection (ai-review.ts:823) already builds a prompt section describing which files in the diff look test-related — a natural building block for a test-generation prompt, not something to duplicate.
parseModelReview (ai-review.ts:645) is the existing pattern for turning raw model text into a typed, validated result; a new parseTestGenResponse-equivalent should follow the same discipline (a strict parser that fails closed to null on malformed output, never throws).
There is no existing "diff → test source" generator anywhere in the codebase — this is the one genuinely new piece of engineering the whole epic needs; everything else in the epic is wiring an existing pattern (fix-handoff, converged-features, PR commands) to a new payload.
Requirements
A pure prompt-composer (e.g. buildE2eTestGenPrompt) that takes the PR's changed-file diffs, the target framework (Playwright — see Epic: AI-generated E2E test coverage for pull requests #4189's epic context on why), and any repo-specific conventions available (e.g. an existing test/ or e2e/ directory listing, existing test file naming patterns) and produces a prompt asking the model for one or more complete Playwright test files.
A strict response parser (e.g. parseE2eTestGenResponse) that extracts fenced code block(s) from the model's response and validates each looks like plausible Playwright source (e.g. contains test(/test.describe(/import { test) before accepting it — malformed or empty output must fail closed to null, exactly like parseModelReview's discipline, never throw and never emit unparseable content into a PR comment.
Model selection must follow the exact same BYOK-vs-default precedence as ai-slop.ts:219-252: the repo's configured BYOK frontier model if present, else a default reviewer-tier model — and the resulting cost event must be labeled byok:<provider> in the BYOK case so it counts toward the existing BYOK daily-repo-limit accounting (DEFAULT_BYOK_DAILY_REPO_LIMIT, ai-review.ts:1079).
Route the actual network call through the existing callAiProvider (ai-review.ts:1178) — no new HTTP/provider-client code.
Cap prompt input size (changed-diff bytes) with a hard ceiling and a documented truncation strategy — an unbounded diff must not silently blow the token budget or the cost estimate.
Unit tests with fixture diffs covering: a clean generation, a malformed/empty model response (must parse to null), a diff with no code files (nothing to generate for), and a BYOK-configured repo (asserting the byok: cost-event label).
Deliverables
buildE2eTestGenPrompt + parseE2eTestGenResponse (naming may vary during implementation), fully unit tested against fixtures.
BYOK-vs-default model selection matching ai-slop.ts's exact precedence and cost-event labeling.
A documented, enforced prompt-size cap.
No new provider-client code — reuses callAiProvider.
Expected outcome
A pure, well-tested "PR diff in, Playwright test source (or null) out" module that #4192 (render) and #4191 (resolver) can call without needing to know anything about prompt construction, model selection, or cost accounting.
Part of #4189. Builds on the config gate in #4190 (no hard code dependency — this is a standalone pure/async module testable against fixture diffs — but ships gated by the same flag).
Context
gittensory already has an established pattern for LLM-generated advisory content that reuses the shared model-calling plumbing in
src/services/ai-review.ts:callAiProvider(ai-review.ts:1178) is the single call site every AI-generated artifact goes through.BEST_REVIEW_MODELS/RELIABLE_FALLBACK_MODELS(ai-review.ts:46-56) are the default model tiers; a repo with BYOK configured uses "the maintainer's own frontier model (Anthropic/OpenAI)" instead — see the exact pattern insrc/services/ai-slop.ts:219,252(model: input.providerKey ? \byok:${input.providerKey.provider}` : WORKERS_SLOP_MODELS.join("+")`), which is how BYOK-vs-default model selection and cost-event labeling already work for the slop advisory writer.estimateNeurons(ai-review.ts:493) andBYOK_MODEL_PRICING_USD_PER_MTOK(ai-review.ts:1097) are the existing cost-accounting primitives.buildTestEvidencePromptSection(ai-review.ts:823) already builds a prompt section describing which files in the diff look test-related — a natural building block for a test-generation prompt, not something to duplicate.parseModelReview(ai-review.ts:645) is the existing pattern for turning raw model text into a typed, validated result; a newparseTestGenResponse-equivalent should follow the same discipline (a strict parser that fails closed tonullon malformed output, never throws).There is no existing "diff → test source" generator anywhere in the codebase — this is the one genuinely new piece of engineering the whole epic needs; everything else in the epic is wiring an existing pattern (fix-handoff, converged-features, PR commands) to a new payload.
Requirements
buildE2eTestGenPrompt) that takes the PR's changed-file diffs, the target framework (Playwright — see Epic: AI-generated E2E test coverage for pull requests #4189's epic context on why), and any repo-specific conventions available (e.g. an existingtest/ore2e/directory listing, existing test file naming patterns) and produces a prompt asking the model for one or more complete Playwright test files.parseE2eTestGenResponse) that extracts fenced code block(s) from the model's response and validates each looks like plausible Playwright source (e.g. containstest(/test.describe(/import { test) before accepting it — malformed or empty output must fail closed tonull, exactly likeparseModelReview's discipline, never throw and never emit unparseable content into a PR comment.ai-slop.ts:219-252: the repo's configured BYOK frontier model if present, else a default reviewer-tier model — and the resulting cost event must be labeledbyok:<provider>in the BYOK case so it counts toward the existing BYOK daily-repo-limit accounting (DEFAULT_BYOK_DAILY_REPO_LIMIT,ai-review.ts:1079).callAiProvider(ai-review.ts:1178) — no new HTTP/provider-client code.null), a diff with no code files (nothing to generate for), and a BYOK-configured repo (asserting thebyok:cost-event label).Deliverables
buildE2eTestGenPrompt+parseE2eTestGenResponse(naming may vary during implementation), fully unit tested against fixtures.ai-slop.ts's exact precedence and cost-event labeling.callAiProvider.Expected outcome
A pure, well-tested "PR diff in, Playwright test source (or null) out" module that #4192 (render) and #4191 (resolver) can call without needing to know anything about prompt construction, model selection, or cost accounting.
Resources / examples
src/services/ai-review.ts:46-56, 493, 645, 823, 1079, 1097, 1178(models, cost estimation, response parsing, test-evidence prompt section, BYOK pricing table, the shared provider call)src/services/ai-slop.ts:219, 252(the exact BYOK-vs-default model selection + cost-event labeling pattern to mirror)Effort
L — the one genuinely new piece of engineering in the epic; everything downstream just wires this module's output into existing patterns.