Skip to content

feat(review): LLM core to turn a PR diff into Playwright E2E test source #4191

Description

@JSONbored

Part of #4189. Builds on the config gate in #4190 (no hard code dependency — this is a standalone pure/async module testable against fixture diffs — but ships gated by the same flag).

Context

gittensory already has an established pattern for LLM-generated advisory content that reuses the shared model-calling plumbing in src/services/ai-review.ts:

  • callAiProvider (ai-review.ts:1178) is the single call site every AI-generated artifact goes through.
  • BEST_REVIEW_MODELS / RELIABLE_FALLBACK_MODELS (ai-review.ts:46-56) are the default model tiers; a repo with BYOK configured uses "the maintainer's own frontier model (Anthropic/OpenAI)" instead — see the exact pattern in src/services/ai-slop.ts:219,252 (model: input.providerKey ? \byok:${input.providerKey.provider}` : WORKERS_SLOP_MODELS.join("+")`), which is how BYOK-vs-default model selection and cost-event labeling already work for the slop advisory writer.
  • estimateNeurons (ai-review.ts:493) and BYOK_MODEL_PRICING_USD_PER_MTOK (ai-review.ts:1097) are the existing cost-accounting primitives.
  • buildTestEvidencePromptSection (ai-review.ts:823) already builds a prompt section describing which files in the diff look test-related — a natural building block for a test-generation prompt, not something to duplicate.
  • parseModelReview (ai-review.ts:645) is the existing pattern for turning raw model text into a typed, validated result; a new parseTestGenResponse-equivalent should follow the same discipline (a strict parser that fails closed to null on malformed output, never throws).

There is no existing "diff → test source" generator anywhere in the codebase — this is the one genuinely new piece of engineering the whole epic needs; everything else in the epic is wiring an existing pattern (fix-handoff, converged-features, PR commands) to a new payload.

Requirements

  • A pure prompt-composer (e.g. buildE2eTestGenPrompt) that takes the PR's changed-file diffs, the target framework (Playwright — see Epic: AI-generated E2E test coverage for pull requests #4189's epic context on why), and any repo-specific conventions available (e.g. an existing test/ or e2e/ directory listing, existing test file naming patterns) and produces a prompt asking the model for one or more complete Playwright test files.
  • A strict response parser (e.g. parseE2eTestGenResponse) that extracts fenced code block(s) from the model's response and validates each looks like plausible Playwright source (e.g. contains test(/test.describe(/import { test) before accepting it — malformed or empty output must fail closed to null, exactly like parseModelReview's discipline, never throw and never emit unparseable content into a PR comment.
  • Model selection must follow the exact same BYOK-vs-default precedence as ai-slop.ts:219-252: the repo's configured BYOK frontier model if present, else a default reviewer-tier model — and the resulting cost event must be labeled byok:<provider> in the BYOK case so it counts toward the existing BYOK daily-repo-limit accounting (DEFAULT_BYOK_DAILY_REPO_LIMIT, ai-review.ts:1079).
  • Route the actual network call through the existing callAiProvider (ai-review.ts:1178) — no new HTTP/provider-client code.
  • Cap prompt input size (changed-diff bytes) with a hard ceiling and a documented truncation strategy — an unbounded diff must not silently blow the token budget or the cost estimate.
  • Unit tests with fixture diffs covering: a clean generation, a malformed/empty model response (must parse to null), a diff with no code files (nothing to generate for), and a BYOK-configured repo (asserting the byok: cost-event label).

Deliverables

  • buildE2eTestGenPrompt + parseE2eTestGenResponse (naming may vary during implementation), fully unit tested against fixtures.
  • BYOK-vs-default model selection matching ai-slop.ts's exact precedence and cost-event labeling.
  • A documented, enforced prompt-size cap.
  • No new provider-client code — reuses callAiProvider.

Expected outcome

A pure, well-tested "PR diff in, Playwright test source (or null) out" module that #4192 (render) and #4191 (resolver) can call without needing to know anything about prompt construction, model selection, or cost accounting.

Resources / examples

  • src/services/ai-review.ts:46-56, 493, 645, 823, 1079, 1097, 1178 (models, cost estimation, response parsing, test-evidence prompt section, BYOK pricing table, the shared provider call)
  • src/services/ai-slop.ts:219, 252 (the exact BYOK-vs-default model selection + cost-event labeling pattern to mirror)
  • Playwright test API docs for validating what "plausible Playwright source" should contain

Effort

L — the one genuinely new piece of engineering in the epic; everything downstream just wires this module's output into existing patterns.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions