Failed to get all albums

[alienff]

The bug

When using a cli to upload photos into album, I often see an error "Failed to get all albums". The full response looks like this:

{"message":"Failed to get all albums","error":"Internal Server Error","statusCode":500,"correlationId":"q50rr9fr"}

Also from logs:

[Nest] 59  - 10/14/2024, 9:13:13 AM   ERROR [Api:ErrorInterceptor~q50rr9fr] Unknown error: Error: timeout exceeded when trying to connect
[Nest] 59  - 10/14/2024, 9:13:13 AM   DEBUG [Api:LoggingInterceptor~q50rr9fr] GET /api/albums 200 15347.24ms ::ffff:192.168.178.93
[Nest] 59  - 10/14/2024, 9:13:13 AM   DEBUG [Api:GlobalExceptionFilter~q50rr9fr] HttpException(500): {"message":"Failed to get all albums","error":"Internal Server Error","statusCode":500}

I tried to run

TIMEFORMAT=" %R"
for i in `seq 1 100` ; do time curl -s -o /dev/null -w "%{http_code}" -H 'X-Api-Key: *****' 'http://127.0.0.1:2283/api/albums' ; done

to get more accurate statistics, and it showed ~80% success rate. Raw data here.

The OS that Immich Server is running on

Ubuntu 24.04.1 LTS

Version of Immich Server

v1.117.0

Version of Immich Mobile App

N/A

Platform with the issue

• Server
• Web
• Mobile
• CLI

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Zurich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=*****

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Create > 280 albums
2. Have a not-so fast machine with 4 GiB of RAM and an Intel Atom.
3. Run

TIMEFORMAT=" %R"
for i in `seq 1 100` ; do time curl -s -o /dev/null -w "%{http_code}" -H 'X-Api-Key: *****' 'http://127.0.0.1:2283/api/albums' ; done

Relevant log output

No response

Additional information

No response

[alienff]

My thoughts.

1. "Unknown error" is probably coming from logger.ts. I am no ts expert, but looks like error logger could log error as an exception rather than as a string. At least server can have a debug setting to print full stack traces instead of just error messages.
2. "Error: timeout exceeded when trying to connect" probably means that the server could not connect to the database. Not sure why it tries to connect on each request, I would expect some persistent connection for faster processing. But maybe this is how node.js works?.. Anyway, a longer (configurable?) timeout can probably help. Or a retry mechanism.
3. getAllAlbums is called just to get an id of a required album. Why not have a server method for that purpose? Seems like a waste of time to pass all albums just to get one.
4. If nothing else, have a retry around "getAllAlbums" on the client side. It is very annoying to retry the whole upload command which although does not really re-upload files, but the verification still takes a long time.

I am willing to implement some mitigation for the issue.

[jrasm91]

There is a connection timeout because the database server is no longer responding to any requests. There are persistent connections in a pool on the server, but they are failing to communicate because your computer has limited CPU cycles available and they are being eaten up by other processes. Likely transcoding, machine learning, thumbnail generation, and other jobs.

[alienff]

Thank you for a reply. What do you think is the best way to mitigate the issue? (Not counting buying a new hardware :) )

[jrasm91]

I would probably recommend reducing or pausing the jobs while you are uploading lots of new images. And, in general, setting pretty low concurrency for the jobs, specifically the machine learning ones, thumbnails, and transcoding.

[alienff]

Setting concurrency to lower settings did not help. In fact, I mostly get failures when there are no running jobs.
I ended up moving immich server to a faster VPS which improved response rate a lot. But I am still not convinced that the error is only because of slow CPU. Main reason is that the only error I have seen is for method getAllAlbums. All other methods worked fine even in my initial setup.

[alienff]

Can I unclose the issue?..