Run the worker in e2e and verify a submitted job is processed#471
Merged
Conversation
Start the worker alongside web in the e2e Docker Compose run, pointed at the no-network stub provider so it makes no paid API calls. A new Playwright spec submits a prompt via POST /api/v1/prompts and polls Postgres until the worker records the prompt_run. Gate the worker's recurring cron schedules behind WORKER_DISABLE_SCHEDULES (default off, so production is unchanged). schedule-maintenance treats any prompt with no run for the configured model as immediately overdue, so without this the worker would re-enqueue the seeded fixtures and mutate data the other specs assert on.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This comment has been minimized.
This comment has been minimized.
Replaces WORKER_DISABLE_SCHEDULES — a test-only flag in production worker code — with test orchestration. The fixture-dependent Playwright and Bruno suites run first with the worker down, then the worker starts and a tagged @worker spec runs on its own. Starting the worker only after those suites finish keeps its self-healing scheduler from re-enqueuing the seeded prompts and mutating data they assert on, with no production code path. The worker now runs exactly as in prod; only its SCRAPE_TARGETS is overridden to the no-network stub provider to avoid paid API calls.
seed.ts ran seed() — a destructive DELETE + re-insert — as a top-level side effect, so importing it for its exported fixture constants re-ran the whole seed. Playwright's globalSetup (auth-setup) and the worker spec both import it, so loading them wiped rows out from under running tests: the worker spec's submitted prompt and its recorded run were deleted mid-poll, and a concurrent wipe made POST /api/v1/prompts return 400 'Brand default not found' on retry. seed() now runs only when the file is the process entry (tsx seed.ts). The worker spec no longer imports seed — it hardcodes the brand id like the other specs.
seed.ts goes back to being a pure script with an unconditional top-level seed() call: with the constants in a side-effect-free module there is nothing left to import from the seeder, so the realpath entry-point guard (which silently skipped seeding on a loader/argv quirk) is unnecessary.
The phase boundary moves out of CI-only grep flags into the Playwright config, so `pnpm test:e2e` runs only the fixture specs again — a bare local run no longer times out against a stopped worker or feeds a paid job to a real one. CI selects --project=worker explicitly for phase 2. The worker project writes to its own outputDir (a second playwright run wipes the output dir it uses, which was deleting phase 1 traces and the Bruno reports from test-results/ before upload) and phase 2 keeps the github reporter for PR annotations while still skipping html so it doesn't overwrite phase 1's report. The readiness log-grep loop is gone: the submitted job is durable in pg-boss, so the spec's poll — now expect.poll with a budget covering worker startup plus one 60s retry cycle — gives the same guarantee, and worker logs dump in a failure() step instead of an inline || wrapper.
Every docker compose step was repeating the elmo.yaml + worker-override pair; a future step that forgot the second -f would silently bring the worker up with real SCRAPE_TARGETS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The e2e suite previously started only
weband deliberately skipped the worker. Now that we have a no-networkstubprovider, this runs the worker in e2e and adds a spec that proves a submitted job is actually processed end-to-end.How it works
The run is split into two phases so the worker's self-healing scheduler can't mutate the seeded fixtures the other tests assert on. The phase boundary lives in the Playwright config as two projects (
fixtures/worker), so every entrypoint inherits it — not just the CI YAML.Phase 1 — worker down. Seed, then run every fixture-dependent suite: the
fixturesPlaywright project and the Bruno API suite. With no worker running, nothing can re-enqueue or reprocess the seeded prompts.pnpm test:e2eruns only this project, so a bare local run never depends on worker topology (and can't feed a paid job to a real worker).Phase 2 — worker up. Start the worker and run
--project=worker. Its one specPOSTs to/api/v1/prompts(which enqueues an immediateprocess-promptjob) andexpect.polls Postgres until aprompt_runsrow with the stub's version appears for the new prompt. There is no separate readiness wait: the job is durable in pg-boss, so the 120s poll budget absorbs worker startup and covers one 60s retry cycle of the queue. Asserting the DB side-effect — not just job state — means it only passes if the handler actually did its work.The
workerproject writes to its ownoutputDirand drops thehtmlreporter (keepinggithubfor PR annotations), so the second Playwright invocation can't wipe phase 1's traces, report, or the Bruno JUnit/JSON reports before the artifact uploads.The worker runs exactly as it does in production; only its
SCRAPE_TARGETSis overridden tostub:stubviae2e/worker-override.yaml. The override is worker-only because the web app derives its model filter/settings UI fromSCRAPE_TARGETS, so web keeps the real value from.env. The compose file set is pinned once via a job-levelCOMPOSE_FILEso every step runs against the same effective config.Shared fixture constants (DB URL, brand/prompt IDs, the test API key) live in a side-effect-free
e2e/fixtures.ts; specs and auth-setup import from it, andseed.tsstays a pure script.Why phased, not a flag
An earlier revision added a
WORKER_DISABLE_SCHEDULESenv branch to the production worker so itsschedule-maintenancecron wouldn't reprocess seeded prompts (that maintenance pass counts any prompt with no run for the configured model as immediately overdue, so it would re-enqueue the run-less fixtures the moment it fired). Gating production code purely for tests is an anti-pattern — this instead orders the run so the fixture suites finish before the worker exists. No production code changes.Verification
tsc --noEmitgreen one2e/; workflow YAML parses.playwright test --listconfirms the split: 21 tests in 5 files underfixtures, exactly the 1 worker spec underworker.total gte 5,totalPages gt 1,prompts[0].brandId) are unaffected, and they run before the worker starts regardless.pnpm test:e2e-local/ act) — not run locally since it needs the Docker +elmo init --devflow.No changeset: internal test/CI plumbing, not user-facing.