Errors in logs

[xttlegendapi]

The bug

There seem to be a lot of errors in the logs when it's doing microservices.

The OS that Immich Server is running on

Ubuntu 20.04 Docker

Version of Immich Server

v1.111.0

Version of Immich Mobile App

v1.111.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - share_B_immich:/usr/src/app/upload
      - share_B_library:/mnt/media/library:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. start some jobs in http://immich_server:3001/admin/jobs-status
2. See the errors

Relevant log output

Here are some examples of the errors

immich_server                | [Nest] 6  - 08/12/2024, 11:39:54 AM   ERROR [Microservices:JobService] Unable to run job handler (metadataExtraction/queue-metadata-extraction): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:39:54 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:39:54 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "force": false
immich_server                | }
immich_server                |

immich_server                | [Nest] 6  - 08/12/2024, 11:41:25 AM   ERROR [Microservices:JobService] Unable to run job handler (sidecar/sidecar-discovery): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:41:25 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:41:25 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "id": "79c118f2-575a-49c9-987c-2c3826ec0b1f"
immich_server                | }
immich_server                |

immich_server                | [Nest] 6  - 08/12/2024, 11:42:13 AM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/smart-search): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:42:13 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:42:13 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "id": "02348eaf-8711-4e5f-8858-3b9a3845cf80"
immich_server                | }
immich_server                |

immich_server                | [Nest] 6  - 08/12/2024, 11:42:53 AM   ERROR [Microservices:JobService] Unable to run job handler (duplicateDetection/queue-duplicate-detection): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:42:53 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:42:53 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "force": false
immich_server                | }
immich_server                |

immich_server                | [Nest] 6  - 08/12/2024, 11:43:06 AM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:43:06 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:43:06 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "id": "a55e688b-ddea-4c31-8d99-a66b7ca67f29"
immich_server                | }
immich_server                |

immich_server                | [Nest] 6  - 08/12/2024, 11:43:18 AM   ERROR [Microservices:JobService] Unable to run job handler (videoConversion/video-conversion): Error: Connection terminated due to connection timeout
immich_server                | [Nest] 6  - 08/12/2024, 11:43:18 AM   ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout
immich_server                |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_server                |     at Object.onceWrapper (node:events:633:28)
immich_server                |     at Connection.emit (node:events:519:28)
immich_server                |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_server                |     at Socket.emit (node:events:519:28)
immich_server                |     at TCP.<anonymous> (node:net:339:12)
immich_server                | [Nest] 6  - 08/12/2024, 11:43:18 AM   ERROR [Microservices:JobService] Object:
immich_server                | {
immich_server                |   "id": "8432b5b2-ffbd-4f71-8954-0e3900fd96a3"
immich_server                | }
immich_server                |

Basicly the only thing I see in the logs are these errors.

As it seems database related this are the postgres logs:

immich_postgres  |
immich_postgres  | PostgreSQL Database directory appears to contain a database; Skipping initialization
immich_postgres  |
immich_postgres  | 2024-08-12 11:37:15.134 UTC [1] LOG:  redirecting log output to logging collector process
immich_postgres  | 2024-08-12 11:37:15.134 UTC [1] HINT:  Future log output will appear in directory "log".

and these are the last log file in the postgres logs directory

2024-08-12 11:37:15.134 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-08-12 11:37:15.135 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-08-12 11:37:15.135 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-08-12 11:37:15.141 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-08-12 11:37:15.149 UTC [27] LOG:  database system was shut down at 2024-08-12 11:37:01 UTC
[2024-08-12T11:37:15Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/511406".
[2024-08-12T11:37:15Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331".
[2024-08-12T11:37:15Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331/segments/6e311503-e4e6-4420-b4ad-f57e3972a35e".
2024-08-12 11:37:15.175 UTC [1] LOG:  database system is ready to accept connections
[2024-08-12T11:37:15Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/511406/segments/955eb484-1bc4-4b35-ad65-59405da61085".

Additional information

No response

[xttlegendapi]

Here are some more of the errors I'm seeing. Is it normal to have so many errors for microservices, its just spamming my logs when I execute jobs, maybe I can understand that some old photos with weird encodings don't work with some of the microservices jobs, but it also seems like the database connection has a problem..? Is there a way to debug whats happening here?

ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: Asset 6ce55f07-07d8-4a07-9698-084ad6b35107 dimensions are unknown
ERROR [Microservices:JobService] Error: Asset 6ce55f07-07d8-4a07-9698-084ad6b35107 dimensions are unknown

ERROR [Microservices:JobService] Unable to run job handler (metadataExtraction/queue-metadata-extraction): Error: Connection terminated due to connection timeout
ERROR [Microservices:JobService] Error: Connection terminated due to connection timeout

 ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: extract_area: bad extract area
ERROR [Microservices:JobService] Error: extract_area: bad extract area

ERROR [Microservices:JobService] Error: ffprobe exited with code 1

[xttlegendapi]

I don't think it can be system overload, my system is at less then 50% of memory usage and less then 30% cpu usage.

[welpher]

Me too.
Version of Immich Server
v1.122.2