[Help Needed] Upload failures (503) and Traefik routing errors with official docker-compose.override.yml.example configuration

[benmilan26]

Hello,

networking, docker and selfhosting itself are unfamiliar fields for myself. I came so far with trial and error and some help of a LLM (lumo by proton and qwen 2.5). Installation and this report are coauthored by AI.

I am experiencing reaccuring issues with my picpeak installation and need some help on how to proceed. I followed the official documentation as far as my understanding allowed me to.

My Environment:

OS: Ubuntu (Infomaniak VPS 2vCPU 2GB RAM, latest version)
Docker Compose: v5.1.4
Traefik: v3.6 (external proxy in network traefik_proxy)
PicPeak: Stable Channel (ghcr.io/the-luap/picpeak/backend:stable)

The Problem:

1. Intermittent Upload Failures: When uploading photos, the first batch often succeeds, but subsequent uploads fail in the browser with the message "Some files could not be uploaded". Sometimes after composing picpeak the first few uploads work, afterwards I get the message again with no or only 1 picture uploaded.
2. 503 Service Unavailable: In the frontend logs, I see 503 status codes when fetching thumbnails (GET /api/admin/photos/{id}/thumbnail/{id}).
3. Traefik Routing Errors: The Traefik logs show repeated errors about service linking.

Configuration Used:

I followed the official documentation and used docker-compose.override.yml.example as a template for my docker-compose.override.yml. This includes:

• Two separate Traefik services defined: picpeak-api and picpeak-media
• Both services pointing to port 3001 (as per documentation)
• Backend container listening internally on port 3000 (verified via health check)

Traefik Errors:

ERR Router picpeak-api cannot be linked automatically with multiple Services: ["picpeak-api" "picpeak-media"]
ERR Router picpeak-media cannot be linked automatically with multiple Services: ["picpeak-api" "picpeak-media"]
ERR error="the service \"picpeak-media@docker\" does not exist"

Frontend Errors:

172.18.0.2 - - [31/May/2026:11:40:08 +0000] "GET /api/admin/photos/3/thumbnail/27 HTTP/1.1" 503 50
...
172.18.0.2 - - [31/May/2026:11:40:22 +0000] "GET /api/admin/photos/3/thumbnail/28 HTTP/1.1" 503 53

What I Have Verified:

• Backend Health: docker exec picpeak-backend wget http://localhost:3000/health returns {"status":"ok"}.
• Port 3001: Does not exist inside the container (docker exec picpeak-backend wget http://localhost:3001/health returns Connection refused).
• Port 3000: Is the correct internal port for the backend.
• Network Connectivity: Traefik can reach the backend container on port 3000 (docker exec traefik wget http://picpeak-backend:3000/health works).
• File Name: Initially had docker-compose.override.yml. (with trailing dot), which I corrected to docker-compose.override.yml.
• Ports in Labels: Currently set to 3001 as per documentation.

My Questions:

Is the port 3001 in the documentation correct, or should it be 3000 for traefik service labels? (suggested by Lumo and qwen)
Is defining two separate services (picpeak-api and picpeak-media) in the docker-compose.override.yml.example intended for a single backend container setup? (added suggestion bei lumo by proton)
What is the recommended configuration for Traefik v3.6 with the PicPeak stable channel?
Should I modify the docker-compose.override.yml to use a different service structure, or is there a configuration issue I am missing?

Attached Files:

• docker-compose.production.yml (modified, sensitive data removed)
• docker-compose.override.yml (based on official example)
• traefik_error_logs.txt (Traefik errors)
• backend_logs.txt
• frontend_logs.txt
• verification_logs.txt (Port verification tests)

I appreciate any guidance on how to proceed. Thank you for your help!
Benedikt

added:
Observed Behavior (Intermittent Failure): The issue is not consistent; it follows a distinct pattern:

Initial Success: Immediately after starting the containers or after a period of inactivity, the first few upload attempts succeed, and thumbnails load correctly.
Degradation: After a short period (usually after processing 1-2 batches of images), the system begins to fail.
Failure Mode: Subsequent uploads result in the browser error "Some files could not be uploaded," and thumbnail requests return 503 Service Unavailable.
Temporary Recovery: Sometimes, waiting a few minutes or refreshing the page temporarily restores functionality for a short time, but the cycle repeats.

docker-compose.production.yml
docker-compose.override.yml
traefik_error_logs.txt
verification_logs.txt
frontend logs.txt
backend_logs.txt

[Luca-Timo]

Hi @benmilan26 — first off, apologies for the two-week silence. We didn't see this: we work mostly out of the Issues tab and the Discussions thread slipped past us. Thanks for your patience, and for the detailed report — there are actually two separate things going on here.

1. Traefik routing

You don't route to the backend at all, and you don't need two services. The frontend container (nginx) is the single entry point — it serves the app and already reverse-proxies /api, /photos, /thumbnails, /uploads, etc. to the backend internally. So point Traefik at the frontend service on port 80 only:

services:
  frontend:
    networks: [traefik, default]          # must share Traefik's network
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.routers.picpeak.rule=Host(`gallery.example.com`)"
      - "traefik.http.routers.picpeak.entrypoints=websecure"
      - "traefik.http.routers.picpeak.tls.certresolver=letsencrypt"
      - "traefik.http.services.picpeak.loadbalancer.server.port=80"
networks:
  traefik:
    external: true

Remove all Traefik labels from the backend service and delete the picpeak-api / picpeak-media services — that's what triggers both "cannot be linked automatically with multiple Services" and "service … does not exist". On the 3001 question: that's only the host-side debug port mapping (${BACKEND_PORT:-3001}:3000); inside Docker the backend listens on 3000, but with the setup above Traefik never needs to know that. You can also drop the FRONTEND_PORT host mapping once Traefik reaches it over the shared network.

2. The intermittent 503s after a batch or two

That part is almost certainly RAM. With 2 vCPU / 2 GB, image processing (sharp/libvips) plus Postgres + Redis + Node can exhaust memory; the backend gets OOM-killed, the health check fails, and Traefik returns 503 until it's back — which matches the "works, then degrades, then recovers" pattern. To confirm:

docker stats
docker inspect picpeak-backend --format '{{.State.OOMKilled}}'
dmesg | grep -i oom

Mitigations: add swap, upload in smaller batches, and ideally bump to ≥4 GB.

Could you share docker compose logs backend around a failed upload? That'll confirm the OOM path. We'll also add a proper reverse-proxy guide to the docs — this should have been documented.

[benmilan26]

Hi @Luca-Timo ,

Thanks for the reply and the clear diagnosis - it is highly appreciated!

1. Traefik Routing (Solved)

Following your advice to remove the specific Traefik labels from the backend. I can confirm that the initial routing errors (ERR Router ... cannot be linked) are gone.

2. The intermittent 503s after a batch or two

During testing with batches of images, two distinct issues came up in the logs:

(coauthored by AI):

• Issue A: Intermittent "400 Bad Request" on Upload Attempts Before any processing starts, the upload requests often fail immediately with a 400 Bad Request. In one test sequence, the first three attempts failed, and only the fourth succeeded (202 Accepted). The frontend logs show warnings about buffering request bodies to temporary files during these failures.
• Issue B: Silent Backend Failures leading to "503 Service Unavailable" After a successful upload (202 Accepted), subsequent requests for thumbnails return 503 Service Unavailable. The backend container does not output any error messages or stack traces before this happens; it simply stops responding. My system monitor shows high memory usage and swap activity during these moments.

Attached: Below are sanitized log excerpts showing the sequence of events (400 errors followed by 503 errors). All file names, IPs, and internal IDs have been anonymized for privacy.

picpeak-frontend | [IP] - - [17/Jun/2026:13:15:47 +0000] "POST /api/admin/[EVENT_ID]/upload HTTP/1.1" 400 0 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | 2026/06/17 13:15:47 [warn] 8#8: *1538 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000018, client: [IP], server: localhost, request: "POST /api/admin/[EVENT_ID]/upload HTTP/1.1", host: "domain.com", referrer: "https://domain.com/admin/[EVENT_ID]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:16:47 +0000] "POST /api/admin/[EVENT_ID]/upload HTTP/1.1" 400 0 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | 2026/06/17 13:16:48 [warn] 8#8: *1999 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000019, client: [IP], server: localhost, request: "POST /api/admin/[EVENT_ID]/upload HTTP/1.1", host: "domain.com", referrer: "https://domain.com/admin/[EVENT_ID]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:17:48 +0000] "POST /api/admin/[EVENT_ID]/upload HTTP/1.1" 400 0 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | 2026/06/17 13:17:49 [warn] 8#8: *2006 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000020, client: [IP], server: localhost, request: "POST /api/admin/[EVENT_ID]/upload HTTP/1.1", host: "domain.com", referrer: "https://domain.com/admin/[EVENT_ID]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:18:04 +0000] "POST /api/admin/[EVENT_ID]/upload HTTP/1.1" 202 975 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:18:05 +0000] "GET /api/admin/photos/[EVENT_ID]/thumbnail/[ID_1] HTTP/1.1" 503 50 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:18:05 +0000] "GET /api/admin/photos/[EVENT_ID]/thumbnail/[ID_2] HTTP/1.1" 503 50 "[URL]" "[Browser-Agent]" "[IP]"
picpeak-frontend | [IP] - - [17/Jun/2026:13:18:05 +0000] "GET /api/admin/photos/[EVENT_ID]/thumbnail/[ID_3] HTTP/1.1" 503 50 "[URL]" "[Browser-Agent]" "[IP]"

picpeak-backend | Temp filename: temp_[TIMESTAMP]_[RANDOM].jpeg
picpeak-backend | Multer destination called for file: [FILE_1].jpeg
picpeak-backend | Temp destination path: /app/storage/temp/upload_[TIMESTAMP]_[RAND]
picpeak-backend | Multer filename called for file: [FILE_1].jpeg
... (repeats for multiple files) ...
picpeak-backend | Upload request received for event: [EVENT_ID]
picpeak-backend | Body: [Object: null prototype] {}
picpeak-backend | Files: 30
picpeak-backend | File details: [
picpeak-backend |   { name: '[FILE_1].jpeg', size: 3240162, mimetype: 'image/jpeg' },
picpeak-backend |   { name: '[FILE_2].jpeg', size: 692271, mimetype: 'image/jpeg' },
picpeak-backend |   { name: '[FILE_3].jpeg', size: 894203, mimetype: 'image/jpeg' },
... (The list of file metadata continues; no error messages) ...
picpeak-backend | Category ID received: undefined
picpeak-backend | Multer destination called for file: [FILE_N].jpeg

Additional Test: Reduced Batch Size

To isolate whether batch size was the trigger for the 400 errors, I performed a test with smaller batches (10 and 17 images).

Result:

No 400 errors occurred in this test. The uploads were accepted immediately (202).
503 errors during thumbnail retrieval persisted. Even with fewer files, the frontend returned Service Unavailable for most thumbnail requests right after upload.
Eventually, some thumbnails succeeded (200) a few seconds later, suggesting the backend recovered temporarily.
Backend logs show no errors during this period. The backend receives the files correctly but does not log any processing steps or completion messages before stopping its output.
Sequence observed:
Upload 202 → Thumbnail Requests 503 (multiple) → ... → Some Thumbnails 200

Could you help me out to understand my problem and to solve it? Is this just a resource-related problem?

Thanks for helping me out

Best regards,
Benedikt

[the-luap]

Hi Benedikt — picking this up. Both new failure modes are the same root cause Luca named (RAM), just presenting as 400 in one window and 503 in the other. Nothing else is going on.

Why 400 in batch A, 503 in batch B

Both come from the backend getting OOM-killed by the upload-processor. The difference is when relative to the request:

• Issue A (400s, large batch) — the previous batch's photos are still being processed in the background when you start a new upload. The backend gets OOM-killed mid-flight, restart: unless-stopped brings it back, and any uploads that arrive during the restart window get rejected. multer sees a partial multipart from a connection that closed mid-stream and returns 400. The nginx "buffered to a temporary file" line is just informational (large bodies always spool to disk past client_body_buffer_size); it's not the cause. Smaller batches (your 10/17-image test) keep peak RSS lower, the backend doesn't restart mid-upload, the 400s disappear.
• Issue B (503s on thumbnails, persists with small batches) — the upload itself completes (202), but the background thumbnail generation of those photos drives memory past the OOM line. The backend dies silently — no stack trace because it's the kernel doing the killing, not Node — and Traefik returns 503 on the thumbnail requests until the container comes back. The "eventually some thumbnails 200" is the recovered backend catching up.

So both halves point to memory headroom during sharp/libvips decode, not request shape or upload size.

Confirming this in 30 seconds

Luca already gave you the three commands; can you run them on the next failure? They're conclusive — no guessing:

# 1 = backend was killed by the kernel
docker inspect picpeak-backend --format '{{.State.OOMKilled}}'

# kernel ring buffer — "out of memory" / "Killed process" lines
sudo dmesg | grep -i -E 'killed|out of memory'

# watch this while uploading a batch
docker stats picpeak-backend

If OOMKilled=true and dmesg shows a "Killed process" line naming node, that's the case closed. If neither — there's a second issue and we keep digging.

What to do right now (still on stable v3.44)

In your .env, add:

UPLOAD_PROCESSOR_CONCURRENCY=1

Restart with docker compose up -d. This drops the background worker loop from 2 to 1 (so only one sharp decode runs at a time), which roughly halves peak RSS during batch processing. Throughput trade-off: a 100-photo batch takes about twice as long, but it won't OOM-kill the backend on 2 GB.

Additionally:

• Add 2-4 GB of swap to the VPS (fallocate -l 4G /swapfile; chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile). It's slower than RAM but won't get you killed; lets sharp finish a decode instead of dying mid-frame.
• If Infomaniak lets you bump to 4 GB RAM, that's the cleaner fix. The docs were wrong calling 2 GB the minimum — 4 GB is what the default config actually needs for normal batches.

What's coming on the next stable

PR #630 (merged to beta yesterday, currently v3.63.0-beta.0) addresses this directly:

• Auto-throttle: backend reads os.totalmem() at boot. When < 3 GB and UPLOAD_PROCESSOR_CONCURRENCY is not set, it defaults to 1 with a one-shot warning naming the override env var. You'll get the right behaviour out of the box without having to find this thread first.
• README: bumped the documented minimum 2 GB → 4 GB, with the "Low-memory hosts" recipe (UPLOAD_PROCESSOR_CONCURRENCY=1) and the throughput trade-off spelled out.
• docker-compose.production.yml: ships a commented mem_limit / memswap_limit example you can uncomment so the kernel takes the container instead of the whole host when memory blows.

Release-please will cut the next stable from beta shortly — at that point an upgrade plus the env var will give you the same behaviour automatically.

If you can run those three diagnostic commands once and confirm OOMKilled=true plus a dmesg "Killed process" line, we're done. If not, post the output and we keep going.

Closing the doc gap on the Traefik side too — your report is going to be the example case in the deployment guide.

[benmilan26]

Hey @the-luap,

I completed a diagnostic cycle comparing behavior with and without UPLOAD_PROCESSOR_CONCURRENCY=1.

1. Test Without CONCURRENCY=1 (Default)

Symptom: Uploads frequently failed with 400 Bad Request on batches >15-30 images dependig image size. Thumbnails returned massive 503 errors.
Diagnostics during failure:

$ docker inspect picpeak-backend --format '{{.State.OOMKilled}}'
false
$ sudo dmesg | grep -i "killed"
(no output)

Observation: The backend did not crash or restart. It became unresponsive (SSH timeouts, hanging requests), but the process remained alive.

2. Test With UPLOAD_PROCESSOR_CONCURRENCY=1

Symptom: Stability improved significantly. I successfully uploaded 200+ images across multiple batches. Intermittent 503 errors still occurred at the very end of heavy loads, but no total system freeze.
Diagnostics after final failure:

$ docker inspect picpeak-backend --format '{{.State.OOMKilled}}'
false

Resource Usage: Peak RAM usage was ~1.3GB (~33% of limit) when issues persisted, confirming we are not hitting a hard memory ceiling that triggers a kill.

Conclusion

High concurrency = Event loop starvation → Timeouts (503) / Rejections (400) without crashing.
Low concurrency (=1) = Serial processing allows recovery, proving the issue is throughput/synchronization, not absolute capacity.

Best regards,
Benedikt

[the-luap]

Hi Benedikt — excellent diagnostic. You're right and I was wrong about the mechanism. OOMKilled: false in both runs, dmesg silent, peak RAM at 1.3 GB — none of that fits an OOM-kill story, and the symptoms you describe ("backend stayed alive but became unresponsive — SSH timeouts, hanging requests") are the textbook event-loop-starvation pattern. Your conclusion ("throughput / synchronization, not absolute capacity") matches the data better than my framing did.

Where my fix still helps + what I got wrong

UPLOAD_PROCESSOR_CONCURRENCY=1 works for the reason you described — fewer parallel sharp/libvips decodes → the main event loop gets CPU time slices back → Traefik health checks pass → no more 503 cascade. PR #630's auto-throttle for hosts with < 3 GB RAM is the right fix, but the warning log message + the README copy say "OOM" when they should say "event-loop starvation." I'll fix the framing — your case is going in the docs as the canonical example.

For reference: #630 is currently in beta (v3.69.x-beta.0 series), not yet in stable — release-please cuts the next stable from beta soon. Once it lands, hosts < 3 GB RAM will auto-default to =1 with a clearer warning that explains why.

The residual 503s at the end of heavy loads

Even with =1, you're getting intermittent 503s on the tail of a batch — that means there's a second bottleneck that throttling the upload-processor doesn't fully cover. Two strong candidates I want to narrow down with one more round of data:

1. sharp is also internally concurrent (hardcoded sharp.concurrency(2))

backend/src/services/imageProcessor.js:13 sets sharp.concurrency(2) — meaning EVEN with one worker loop, each decode still spawns up to two pthreads. On 2 vCPUs that's still saturating the box. Setting UPLOAD_PROCESSOR_CONCURRENCY=1 halves the queue but not the per-decode parallelism. We should expose this as a second env var (SHARP_CONCURRENCY=1) for the same auto-throttle treatment — I'll line that up.

2. Nginx (in front of the backend) or Traefik timing out on a stalled response

There are two proxy_read_timeout values in frontend/nginx.conf — 86400 on the upload location, but 60s on the default route. If the thumbnail/photo-fetch routes hit the 60s timeout while sharp is mid-decode, that produces exactly the 503 pattern you see (admin photo grid drops thumbnails on the floor).

What would help nail it

When you next hit a tail-end 503, can you run:

# 1. Did the backend log anything? Look for "ECONNRESET", "timeout",
#    "knex: Timeout acquiring a connection", or a stack trace.
docker compose logs --tail=200 picpeak-backend

# 2. CPU + RAM during the failure — confirms event-loop starvation
#    (the user CPU column will peg at 100% on one of your 2 vCPUs while
#    the rest of the box is idle).
docker stats picpeak-backend --no-stream

# 3. Which Traefik service returned the 503? (entrypoint/path tells us
#    whether it's the upload route or the thumbnail/static route.)
docker compose logs --tail=100 traefik | grep -E '503|gallery|picpeak'

# 4. How big is your Postgres pool actually getting under load?
docker compose exec postgres psql -U picpeak -d picpeak -c \
  "SELECT count(*) FILTER (WHERE state = 'active') AS active, count(*) AS total FROM pg_stat_activity WHERE datname='picpeak';"

If docker stats shows one CPU pegged + RAM nowhere near a limit, that's the smoking gun for event-loop starvation + the sharp-concurrency fix above. If the backend logs show knex: Timeout acquiring a connection, that's a Postgres pool concern and we adjust differently. If neither and the Traefik 503 logs show the thumbnail route specifically, that's the 60s proxy_read_timeout and a config tweak fixes it.

One round of those four outputs + I can either propose a config change for your install (if it's the nginx timeout) or push a fix that lands with the next beta (if it's the sharp concurrency / pool sizing).

Smaller wins regardless of root cause

While we narrow this down:

• The 2-4 GB swap recommendation Luca mentioned is still worth it — it doesn't address starvation but gives the kernel breathing room if something else spikes.
• You can lower UPLOAD_PROCESSOR_CONCURRENCY further (it accepts 0 to fully pause the worker — useful for batch ingestion windows where you upload first and process after).
• The Traefik routing simplification Luca laid out (single route to the frontend's :80, drop the API/media split) closes the routing errors entirely — those are independent of the 503 cause.

Your diagnostic approach has been better than mine on this one; the rigor is appreciated. Once the four outputs land I'll either point at a config tweak or open a follow-up PR.