Skip to content

feat(tasks): image variant preprocessing queue engine (BE-3 parts 1+2)#481

Merged
Zheaoli merged 3 commits into
mainfrom
feat/image-preprocess-queue
May 30, 2026
Merged

feat(tasks): image variant preprocessing queue engine (BE-3 parts 1+2)#481
Zheaoli merged 3 commits into
mainfrom
feat/image-preprocess-queue

Conversation

@Zheaoli
Copy link
Copy Markdown
Collaborator

@Zheaoli Zheaoli commented May 30, 2026

BE-3 part 1 — variant preprocessing processor

Part of the image-performance overhaul (parent task #1). BE-3 (the bulk variant generation / backfill pipeline) is landing in parts for reviewability since it's the largest backend piece:

  • Part 1 (this PR) — types + the per-image processor (generation + upload logic).
  • Part 2 (next) — the queue orchestration: a preprocess-images task on the existing AdminTaskRun framework (cursor batching + lease + advisory lock + tick), mirroring service.ts.
  • Part 3 (next) — backfill entries: a CLI drain script (pnpm run preprocess:backfill) for one-time bulk + a /admin/tasks button.

This PR

  • types/admin-tasks.tsADMIN_TASK_KEY_PREPROCESS_IMAGES (AdminTaskKey is now a union), PreprocessTaskScope ({ force }), and its normalizer.
  • server/tasks/image-preprocess.tspreprocessImage(image, storage, signal):
    • Fetches the original (30s timeout + cancellation), generates responsive variants via BE-2's generateImageVariants, uploads them to BE-4's resolveVariantStorage backend.
    • Uploads tiers smallest-to-largest, advancing ready_max_width only past a tier whose avif AND webp both uploaded → the persisted value always points at a fully-available tier (the loader invariant @Picimpact-Review flagged).
    • Object key = {storageFolder}/{buildVariantKey(image_key, w, fmt)} so the public URL {variantBaseUrl}/{image_key}_{w}.{fmt} resolves (per BE-4 review).
    • variants_ready flips true only when every tier uploaded; partial failure keeps it false (persisting the partial ready_max_width) so the backfill scope re-picks the image — re-upload is idempotent via content-addressed keys.

Verification

  • tsc --noEmit: no errors in the new file. eslint --fix: clean.
  • End-to-end (real fetch + storage upload) will be exercised when part 2's queue drives a real backfill — there is no test harness / storage creds in this environment. The processor is pure-ish (storage client injected) and unit-testable.

🤖 Generated with Claude Code

Zheaoli and others added 2 commits May 30, 2026 19:45
Types-only groundwork for the image preprocessing queue (BE-3): adds the
ADMIN_TASK_KEY_PREPROCESS_IMAGES key (AdminTaskKey is now a union),
PreprocessTaskScope, and its normalizer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-image variant generation + upload logic for the preprocessing pipeline.
server/tasks/image-preprocess.ts:

- preprocessImage(image, storage, signal): fetch the original (timeout +
  cancellation), generate responsive variants via BE-2's
  generateImageVariants, and upload them to the resolved variant storage
  backend (BE-4's resolveVariantStorage).
- Uploads tiers smallest-to-largest and advances ready_max_width only past
  a tier whose avif AND webp objects both uploaded, so the persisted value
  always points at a fully-available tier — the invariant the gallery loader
  relies on.
- Object key = `{storageFolder}/{buildVariantKey(image_key, w, fmt)}` so the
  public URL `{variantBaseUrl}/{image_key}_{w}.{fmt}` resolves (baseUrl
  already includes the folder; image_key includes the `variants/` prefix).
- variants_ready flips true only when every tier uploaded; a partial failure
  keeps it false (and persists the partial ready_max_width) so the backfill
  scope re-picks the image — re-upload is idempotent via content-addressed
  keys.
- Reuses the cancellation primitives from metadata-refresh.

Queue orchestration (AdminTaskRun lifecycle for the preprocess task) and the
backfill entries (CLI + /admin/tasks) follow in subsequent parts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel Bot commented May 30, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
picimpact Ready Ready Preview, Comment May 30, 2026 11:58am

server/tasks/image-preprocess-service.ts — the preprocess-images task on the
existing AdminTaskRun framework, mirroring server/tasks/service.ts (the
reviewed metadata-refresh orchestration) with preprocess-specific bits:

- Own advisory lock id (42012) so it never contends with the metadata task;
  smaller batch (4) + longer lease (5m) since variant generation is heavy;
  images processed sequentially within a batch to bound memory.
- Scope = every non-deleted image, restricted to variants_ready=false unless
  `force` (reprocess all, e.g. after a tier-ladder/encoder change).
- Per image: resolve the variant storage backend once per kick, run
  preprocessImage, and persist updates whenever present (not only on success)
  so partial uploads advance ready_max_width and image_key/dims/blurhash are
  stored even on failure (per BE-3 part-1 review).
- create refuses when variant_storage is unconfigured; kick fails the run
  with a clear error if the backend disappears mid-run.
- Full lifecycle preserved from the proven metadata orchestration: lease,
  per-image checkpoint (advisory-locked) with lease refresh + cooperative
  cancellation, and succeeded/failed/cancelled finalization; cursor pagination
  for resumability. Exports create/kick/tick/cancel/list/detail/previewCount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Zheaoli Zheaoli changed the title feat(tasks): image variant preprocessing processor (BE-3 part 1) feat(tasks): image variant preprocessing queue engine (BE-3 parts 1+2) May 30, 2026
@Zheaoli
Copy link
Copy Markdown
Collaborator Author

Zheaoli commented May 30, 2026

Updated: added part 2 (queue orchestration) to this branch — server/tasks/image-preprocess-service.ts. Folding it in so the PR is the cohesive engine rather than merging an unused processor on its own.

The orchestration is a faithful mirror of the already-reviewed server/tasks/service.ts (metadata task), with preprocess-specific adaptations:

  • Own advisory lock id 42012 (never contends with the metadata task); batch size 4 + lease 5m (variant gen is heavy); sequential within a batch to bound memory.
  • Scope = non-deleted images, variants_ready=false unless force.
  • Per image: resolve storage once per kick → preprocessImagepersist updates whenever present (not only success) per your part-1 feedback (point 1).
  • create refuses when variant_storage unconfigured; kick fails with a clear error if the backend disappears mid-run.
  • Full lifecycle preserved: lease + per-image advisory-locked checkpoint (lease refresh + cooperative cancel) + succeeded/failed/cancelled finalization + cursor resumability. Exports create/kick/tick/cancel/list/detail/previewCount.

tsc --noEmit + eslint clean. The processor portion (part 1) is unchanged from your approval. Reviewing tip: diff image-preprocess-service.ts against service.ts — the lifecycle logic is intentionally identical.

Part 3 (CLI preprocess:backfill + /admin/tasks button + hono endpoints) will follow and is where the real end-to-end backfill test happens.

@Zheaoli Zheaoli merged commit aaf14b5 into main May 30, 2026
6 checks passed
@Zheaoli Zheaoli deleted the feat/image-preprocess-queue branch May 30, 2026 11:58
pull Bot pushed a commit to candies404/PicImpact that referenced this pull request May 30, 2026
Expose the variant preprocessing queue (the engine from besscroft#481) over the
protected /api/v1/preprocess-tasks routes, mirroring /api/v1/tasks:
preview-count, runs, runs/:id, runs (create), runs/:id/kick, runs/:id/cancel,
tick. Delegates to image-preprocess-service.ts; scope carries the `force`
flag; surfaces "not configured" / "already active" / "no images" as 4xx.

This is the drivable API the admin /tasks button and the CLI backfill
(following) use to create and drain a backfill run; it also enables the
real end-to-end backfill test once variant_storage is configured (besscroft#483).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant