feat(tasks): image variant preprocessing queue engine (BE-3 parts 1+2)#481
Conversation
Types-only groundwork for the image preprocessing queue (BE-3): adds the ADMIN_TASK_KEY_PREPROCESS_IMAGES key (AdminTaskKey is now a union), PreprocessTaskScope, and its normalizer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-image variant generation + upload logic for the preprocessing pipeline.
server/tasks/image-preprocess.ts:
- preprocessImage(image, storage, signal): fetch the original (timeout +
cancellation), generate responsive variants via BE-2's
generateImageVariants, and upload them to the resolved variant storage
backend (BE-4's resolveVariantStorage).
- Uploads tiers smallest-to-largest and advances ready_max_width only past
a tier whose avif AND webp objects both uploaded, so the persisted value
always points at a fully-available tier — the invariant the gallery loader
relies on.
- Object key = `{storageFolder}/{buildVariantKey(image_key, w, fmt)}` so the
public URL `{variantBaseUrl}/{image_key}_{w}.{fmt}` resolves (baseUrl
already includes the folder; image_key includes the `variants/` prefix).
- variants_ready flips true only when every tier uploaded; a partial failure
keeps it false (and persists the partial ready_max_width) so the backfill
scope re-picks the image — re-upload is idempotent via content-addressed
keys.
- Reuses the cancellation primitives from metadata-refresh.
Queue orchestration (AdminTaskRun lifecycle for the preprocess task) and the
backfill entries (CLI + /admin/tasks) follow in subsequent parts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
server/tasks/image-preprocess-service.ts — the preprocess-images task on the existing AdminTaskRun framework, mirroring server/tasks/service.ts (the reviewed metadata-refresh orchestration) with preprocess-specific bits: - Own advisory lock id (42012) so it never contends with the metadata task; smaller batch (4) + longer lease (5m) since variant generation is heavy; images processed sequentially within a batch to bound memory. - Scope = every non-deleted image, restricted to variants_ready=false unless `force` (reprocess all, e.g. after a tier-ladder/encoder change). - Per image: resolve the variant storage backend once per kick, run preprocessImage, and persist updates whenever present (not only on success) so partial uploads advance ready_max_width and image_key/dims/blurhash are stored even on failure (per BE-3 part-1 review). - create refuses when variant_storage is unconfigured; kick fails the run with a clear error if the backend disappears mid-run. - Full lifecycle preserved from the proven metadata orchestration: lease, per-image checkpoint (advisory-locked) with lease refresh + cooperative cancellation, and succeeded/failed/cancelled finalization; cursor pagination for resumability. Exports create/kick/tick/cancel/list/detail/previewCount. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Updated: added part 2 (queue orchestration) to this branch — The orchestration is a faithful mirror of the already-reviewed
Part 3 (CLI |
Expose the variant preprocessing queue (the engine from besscroft#481) over the protected /api/v1/preprocess-tasks routes, mirroring /api/v1/tasks: preview-count, runs, runs/:id, runs (create), runs/:id/kick, runs/:id/cancel, tick. Delegates to image-preprocess-service.ts; scope carries the `force` flag; surfaces "not configured" / "already active" / "no images" as 4xx. This is the drivable API the admin /tasks button and the CLI backfill (following) use to create and drain a backfill run; it also enables the real end-to-end backfill test once variant_storage is configured (besscroft#483). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BE-3 part 1 — variant preprocessing processor
Part of the image-performance overhaul (parent task #1). BE-3 (the bulk variant generation / backfill pipeline) is landing in parts for reviewability since it's the largest backend piece:
preprocess-imagestask on the existingAdminTaskRunframework (cursor batching + lease + advisory lock + tick), mirroringservice.ts.pnpm run preprocess:backfill) for one-time bulk + a/admin/tasksbutton.This PR
ADMIN_TASK_KEY_PREPROCESS_IMAGES(AdminTaskKeyis now a union),PreprocessTaskScope({ force }), and its normalizer.preprocessImage(image, storage, signal):generateImageVariants, uploads them to BE-4'sresolveVariantStoragebackend.ready_max_widthonly past a tier whose avif AND webp both uploaded → the persisted value always points at a fully-available tier (the loader invariant @Picimpact-Review flagged).{storageFolder}/{buildVariantKey(image_key, w, fmt)}so the public URL{variantBaseUrl}/{image_key}_{w}.{fmt}resolves (per BE-4 review).variants_readyflips true only when every tier uploaded; partial failure keeps it false (persisting the partialready_max_width) so the backfill scope re-picks the image — re-upload is idempotent via content-addressed keys.Verification
tsc --noEmit: no errors in the new file.eslint --fix: clean.🤖 Generated with Claude Code