immich server crashing sometimes on restart with postgres/pgvecto.rs error

[electric3055]

The bug

Sometimes when starting immich, the immich-server container fails with the following error. It seems that if I retry restarting multiple times it eventually succeeds.

2025-02-12 10:05:40 QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read https://docs.pgvecto.rs/admin/configuration.html.
2025-02-12 10:05:40 at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2025-02-12 10:05:40 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40 at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40 at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40 at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40 at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40 at async /usr/src/app/dist/repositories/database.repository.js:211:23 {
2025-02-12 10:05:40 query: 'SELECT idx_status FROM pg_vector_index_stat WHERE indexname = $1',
2025-02-12 10:05:40 parameters: [ 'clip_index' ],
2025-02-12 10:05:40 driverError: error: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read https://docs.pgvecto.rs/admin/configuration.html.
2025-02-12 10:05:40 at /usr/src/app/node_modules/pg/lib/client.js:535:17
2025-02-12 10:05:40 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40 at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
2025-02-12 10:05:40 at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40 at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40 at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40 at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40 at async /usr/src/app/dist/repositories/database.repository.js:211:23 {
2025-02-12 10:05:40 length: 272,
2025-02-12 10:05:40 severity: 'ERROR',
2025-02-12 10:05:40 code: 'XX000',
2025-02-12 10:05:40 detail: undefined,
2025-02-12 10:05:40 hint: undefined,
2025-02-12 10:05:40 position: undefined,
2025-02-12 10:05:40 internalPosition: undefined,
2025-02-12 10:05:40 internalQuery: undefined,
2025-02-12 10:05:40 where: undefined,
2025-02-12 10:05:40 schema: undefined,
2025-02-12 10:05:40 table: undefined,
2025-02-12 10:05:40 column: undefined,
2025-02-12 10:05:40 dataType: undefined,
2025-02-12 10:05:40 constraint: undefined,
2025-02-12 10:05:40 file: 'error.rs',
2025-02-12 10:05:40 line: '7',
2025-02-12 10:05:40 routine: undefined
2025-02-12 10:05:40 },
2025-02-12 10:05:40 length: 272,
2025-02-12 10:05:40 severity: 'ERROR',
2025-02-12 10:05:40 code: 'XX000',
2025-02-12 10:05:40 detail: undefined,
2025-02-12 10:05:40 hint: undefined,
2025-02-12 10:05:40 position: undefined,
2025-02-12 10:05:40 internalPosition: undefined,
2025-02-12 10:05:40 internalQuery: undefined,
2025-02-12 10:05:40 where: undefined,
2025-02-12 10:05:40 schema: undefined,
2025-02-12 10:05:40 table: undefined,
2025-02-12 10:05:40 column: undefined,
2025-02-12 10:05:40 dataType: undefined,
2025-02-12 10:05:40 constraint: undefined,
2025-02-12 10:05:40 file: 'error.rs',
2025-02-12 10:05:40 line: '7',
2025-02-12 10:05:40 routine: undefined
2025-02-12 10:05:40 }
2025-02-12 10:05:40 microservices worker error: QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read https://docs.pgvecto.rs/admin/configuration.html., stack: QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read https://docs.pgvecto.rs/admin/configuration.html.
2025-02-12 10:05:40 at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2025-02-12 10:05:40 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40 at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40 at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40 at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40 at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40 at async /usr/src/app/dist/repositories/database.repository.js:211:23
2025-02-12 10:05:40 microservices worker exited with code 1
2025-02-12 10:05:40 Killing api process

The OS that Immich Server is running on

Windows 10

Version of Immich Server

v1.125.7

Version of Immich Mobile App

N/A

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
#      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${UPLOAD_LOCATION}:/upload
      - "D:/digital-pictures:/home/user/photos1:ro"
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: no
    healthcheck:
      disable: false
    labels:
      - "com.centurylinklabs.watchtower.monitor-only=true"

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
#    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
#       file: hwaccel.ml.yml
#       service: vaapi-wsl # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable   
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - ./model-cache:/cache
    ports:
      - '3003:3003'
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    ports:
      - 5432:5432
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

#  power-tools:
#    container_name: immich_power_tools
#    image: ghcr.io/varun-raj/immich-power-tools:latest
#    ports:
#      - "2284:3000"
#    env_file:
#      - .env

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=D:/_ImmichUploads
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Israel

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
#IMMICH_VERSION="v1.126.0"

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_HOST=database
DB_PORT=5432

IMMICH_API_KEY=removed # your_immich_api_key
IMMICH_URL = "http://190.168.0.177:2283" # Your immich instace ip address and port
EXTERNAL_IMMICH_URL = "removed" # External address of immich

# Optional
GOOGLE_MAPS_API_KEY= # Google Maps API Key
# GEMINI_API_KEY=removed # Gemini API Key
GEMINI_API_KEY= # Gemini API Key

Reproduction steps

1. Restart immich

...

Relevant log output

2025-02-12 10:05:40 QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2025-02-12 10:05:40     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2025-02-12 10:05:40     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40     at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40     at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40     at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40     at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40     at async /usr/src/app/dist/repositories/database.repository.js:211:23 {
2025-02-12 10:05:40   query: 'SELECT idx_status FROM pg_vector_index_stat WHERE indexname = $1',
2025-02-12 10:05:40   parameters: [ 'clip_index' ],
2025-02-12 10:05:40   driverError: error: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40   ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2025-02-12 10:05:40       at /usr/src/app/node_modules/pg/lib/client.js:535:17
2025-02-12 10:05:40       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40       at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
2025-02-12 10:05:40       at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40       at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40       at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40       at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40       at async /usr/src/app/dist/repositories/database.repository.js:211:23 {
2025-02-12 10:05:40     length: 272,
2025-02-12 10:05:40     severity: 'ERROR',
2025-02-12 10:05:40     code: 'XX000',
2025-02-12 10:05:40     detail: undefined,
2025-02-12 10:05:40     hint: undefined,
2025-02-12 10:05:40     position: undefined,
2025-02-12 10:05:40     internalPosition: undefined,
2025-02-12 10:05:40     internalQuery: undefined,
2025-02-12 10:05:40     where: undefined,
2025-02-12 10:05:40     schema: undefined,
2025-02-12 10:05:40     table: undefined,
2025-02-12 10:05:40     column: undefined,
2025-02-12 10:05:40     dataType: undefined,
2025-02-12 10:05:40     constraint: undefined,
2025-02-12 10:05:40     file: 'error.rs',
2025-02-12 10:05:40     line: '7',
2025-02-12 10:05:40     routine: undefined
2025-02-12 10:05:40   },
2025-02-12 10:05:40   length: 272,
2025-02-12 10:05:40   severity: 'ERROR',
2025-02-12 10:05:40   code: 'XX000',
2025-02-12 10:05:40   detail: undefined,
2025-02-12 10:05:40   hint: undefined,
2025-02-12 10:05:40   position: undefined,
2025-02-12 10:05:40   internalPosition: undefined,
2025-02-12 10:05:40   internalQuery: undefined,
2025-02-12 10:05:40   where: undefined,
2025-02-12 10:05:40   schema: undefined,
2025-02-12 10:05:40   table: undefined,
2025-02-12 10:05:40   column: undefined,
2025-02-12 10:05:40   dataType: undefined,
2025-02-12 10:05:40   constraint: undefined,
2025-02-12 10:05:40   file: 'error.rs',
2025-02-12 10:05:40   line: '7',
2025-02-12 10:05:40   routine: undefined
2025-02-12 10:05:40 }
2025-02-12 10:05:40 microservices worker error: QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`., stack: QueryFailedError: pgvecto.rs: IPC connection is closed unexpected.
2025-02-12 10:05:40 ADVICE: The error is raisen by background worker errors. Please check the full PostgreSQL log to get more information. Please read `https://docs.pgvecto.rs/admin/configuration.html`.
2025-02-12 10:05:40     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2025-02-12 10:05:40     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-02-12 10:05:40     at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
2025-02-12 10:05:40     at async DatabaseRepository.shouldReindex (/usr/src/app/dist/repositories/database.repository.js:150:25)
2025-02-12 10:05:40     at async DatabaseService.checkReindexing (/usr/src/app/dist/services/database.service.js:145:17)
2025-02-12 10:05:40     at async /usr/src/app/dist/services/database.service.js:93:13
2025-02-12 10:05:40     at async /usr/src/app/dist/repositories/database.repository.js:211:23
2025-02-12 10:05:40 microservices worker exited with code 1
2025-02-12 10:05:40 Killing api process

Additional information

No response

[bo0tzz]

Try updating your pgvecto.rs version to 0.3.0