Very High CPU Usage in Immich Server #15793

The-Inamati · 2025-01-30T16:32:28Z

The-Inamati
Jan 30, 2025

Hello,
I am experiencing an issue with Immich that has never happened before (at least to me) and I require your help.

When I have Immich running after a while I get HUGE CPU spikes that basically make the VM unusable.

The VM has 6GB of RAM and 8 Cores so it should be more than enough to run it (and indeed it was up until now).

These are the two processes that seem to cause the issue.

To give you an idea here is the graph collected during the event.

As you can see I even stop getting the metrics from the VM.

Any help is appreciated as I have no idea what is going on. I am running version 1.125.2 and this is my compose file.

secrets:
  immichdb_pass:
    file: $SECRETS_DIR/immichdb_pass

volumes:
  immich:
  immichml:
  immichdb:

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:v1.125.2
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    secrets:
      - immichdb_pass
    networks:
      - traefik
      - database
    volumes:
      - immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    environment:
      REDIS_HOSTNAME: immichred
      DB_HOSTNAME: immichdb
      DB_USERNAME: immich
      DB_DATABASE_NAME: immichdb
      DB_PASSWORD_FILE: /run/secrets/immichdb_pass
    depends_on:
      - immichred
      - immichdb
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.immich.entrypoints=https"
      - "traefik.http.routers.immich.rule=Host(`photos.$DOMAINNAME1`)"
      - "traefik.http.routers.immich.tls=true"
      ## HTTP Services
      - "traefik.http.routers.immich.service=immich"
      - "traefik.http.services.immich.loadbalancer.server.port=2283"
      - "traefik.http.routers.immich.middlewares=no-auth@file"


  immich-machine-learning:
    container_name: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:v1.125.2
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    secrets:
      - immichdb_pass
    networks:
      - database
    environment:
      REDIS_HOSTNAME: immichred
      DB_HOSTNAME: immichdb
      DB_USERNAME: immich
      DB_DATABASE_NAME: immichdb
      DB_PASSWORD_FILE: /run/secrets/immichdb_pass
    volumes:
      - immichml:/cache

  immichred:
    container_name: immichred
    image: docker.io/redis:6.2-alpine@sha256:905c4ee67b8e0aa955331960d2aa745781e6bd89afc44a8584bfd13bc890f0ae
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    networks:
      - database

  immichdb:
    container_name: immichdb
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    security_opt:
      - no-new-privileges:true
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: unless-stopped
    secrets:
      - immichdb_pass
    networks:
      - database
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/immichdb_pass
      POSTGRES_USER: immich
      POSTGRES_DB: immichdb
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - immichdb:/var/lib/postgresql/data

Mraedis · 2025-01-30T16:44:19Z

Mraedis
Jan 30, 2025
Collaborator

Anything in the logs? docker compose logs

0 replies

The-Inamati · 2025-01-30T16:57:08Z

The-Inamati
Jan 30, 2025
Author

I have a lot of these.

immich-server  | Query failed : {
immich-server  |   durationMs: 1.5989479999989271,
immich-server  |   error: PostgresError: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
immich-server  |       at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
immich-server  |       at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
immich-server  |       at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server  |       at Socket.emit (node:events:524:28)
immich-server  |       at addChunk (node:internal/streams/readable:561:12)
immich-server  |       at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server  |       at Readable.push (node:internal/streams/readable:392:5)
immich-server  |       at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server  |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich-server  |     severity_local: 'ERROR',
immich-server  |     severity: 'ERROR',
immich-server  |     code: '23505',
immich-server  |     detail: 'Key ("ownerId", checksum)=(a24bff72-d145-4097-8f30-373f8bd00870, \\x2c2781c89331cc1097f9135ab63884b024b095d0) already exists.',
immich-server  |     schema_name: 'public',
immich-server  |     table_name: 'assets',
immich-server  |     constraint_name: 'UQ_assets_owner_checksum',
immich-server  |     file: 'nbtinsert.c',
immich-server  |     line: '663',
immich-server  |     routine: '_bt_check_unique'
immich-server  |   },
immich-server  |   sql: 'insert into "assets" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "isArchived", "duration", "isVisible", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *',
immich-server  |   params: [
immich-server  |     'a24bff72-d145-4097-8f30-373f8bd00870',
immich-server  |     null,
immich-server  |     <Buffer 2c 27 81 c8 93 31 cc 10 97 f9 13 5a b6 38 84 b0 24 b0 95 d0>,
immich-server  |     'upload/upload/a24bff72-d145-4097-8f30-373f8bd00870/d5/43/d543a95c-0c4a-49ce-af22-c3d6c4da9765.jpg',
immich-server  |     '1000022797',
immich-server  |     'f8a4e947a0f59f9f40303b7da752b5c31dc03c918ac9b93df48e91ed226e231c',
immich-server  |     2024-09-26T18:40:52.000Z,
immich-server  |     2024-09-26T18:40:54.000Z,
immich-server  |     2024-09-26T18:40:52.000Z,
immich-server  |     'IMAGE',
immich-server  |     false,
immich-server  |     false,
immich-server  |     '0:00:00.000000',
immich-server  |     true,
immich-server  |     'IMG20240926194052.jpg'
immich-server  |   ]
immich-server  | }
immich-server  | Query failed : {
immich-server  |   durationMs: 0.9298219999764115,
immich-server  |   error: PostgresError: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
immich-server  |       at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
immich-server  |       at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
immich-server  |       at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server  |       at Socket.emit (node:events:524:28)
immich-server  |       at addChunk (node:internal/streams/readable:561:12)
immich-server  |       at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server  |       at Readable.push (node:internal/streams/readable:392:5)
immich-server  |       at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server  |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich-server  |     severity_local: 'ERROR',
immich-server  |     severity: 'ERROR',
immich-server  |     code: '23505',
immich-server  |     detail: 'Key ("ownerId", checksum)=(a24bff72-d145-4097-8f30-373f8bd00870, \\x7365686004720723c69f6750cc3ae2a6eb4506a5) already exists.',
immich-server  |     schema_name: 'public',
immich-server  |     table_name: 'assets',
immich-server  |     constraint_name: 'UQ_assets_owner_checksum',
immich-server  |     file: 'nbtinsert.c',
immich-server  |     line: '663',
immich-server  |     routine: '_bt_check_unique'
immich-server  |   },
immich-server  |   sql: 'insert into "assets" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "isArchived", "duration", "isVisible", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *',
immich-server  |   params: [
immich-server  |     'a24bff72-d145-4097-8f30-373f8bd00870',
immich-server  |     null,
immich-server  |     <Buffer 73 65 68 60 04 72 07 23 c6 9f 67 50 cc 3a e2 a6 eb 45 06 a5>,
immich-server  |     'upload/upload/a24bff72-d145-4097-8f30-373f8bd00870/3b/9f/3b9f598c-da4a-4df2-8354-f1682144e82e.jpg',
immich-server  |     '1000022798',
immich-server  |     'f8a4e947a0f59f9f40303b7da752b5c31dc03c918ac9b93df48e91ed226e231c',
immich-server  |     2024-09-26T18:41:10.000Z,
immich-server  |     2024-09-26T18:41:11.000Z,
immich-server  |     2024-09-26T18:41:10.000Z,
immich-server  |     'IMAGE',
immich-server  |     false,
immich-server  |     false,
immich-server  |     '0:00:00.000000',
immich-server  |     true,
immich-server  |     'IMG20240926194110.jpg'
immich-server  |   ]
immich-server  | }
immich-server  | Query failed : {
immich-server  |   durationMs: 0.6697849999181926,
immich-server  |   error: PostgresError: duplicate key value violates unique constraint "UQ_assets_owner_checksum"
immich-server  |       at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:788:26)
immich-server  |       at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:474:6)
immich-server  |       at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
immich-server  |       at Socket.emit (node:events:524:28)
immich-server  |       at addChunk (node:internal/streams/readable:561:12)
immich-server  |       at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
immich-server  |       at Readable.push (node:internal/streams/readable:392:5)
immich-server  |       at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
immich-server  |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich-server  |     severity_local: 'ERROR',
immich-server  |     severity: 'ERROR',
immich-server  |     code: '23505',
immich-server  |     detail: 'Key ("ownerId", checksum)=(a24bff72-d145-4097-8f30-373f8bd00870, \\xd96dc8d82ef455eb70a4c27c280a6e263ca5659a) already exists.',
immich-server  |     schema_name: 'public',
immich-server  |     table_name: 'assets',
immich-server  |     constraint_name: 'UQ_assets_owner_checksum',
immich-server  |     file: 'nbtinsert.c',
immich-server  |     line: '663',
immich-server  |     routine: '_bt_check_unique'
immich-server  |   },
immich-server  |   sql: 'insert into "assets" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "isArchived", "duration", "isVisible", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *',
immich-server  |   params: [
immich-server  |     'a24bff72-d145-4097-8f30-373f8bd00870',
immich-server  |     null,
immich-server  |     <Buffer d9 6d c8 d8 2e f4 55 eb 70 a4 c2 7c 28 0a 6e 26 3c a5 65 9a>,
immich-server  |     'upload/upload/a24bff72-d145-4097-8f30-373f8bd00870/44/07/44070129-e05f-412e-ba4a-7a5bf2fbc096.jpg',
immich-server  |     '1000022799',
immich-server  |     'f8a4e947a0f59f9f40303b7da752b5c31dc03c918ac9b93df48e91ed226e231c',
immich-server  |     2024-09-26T18:43:02.000Z,
immich-server  |     2024-09-26T18:43:03.000Z,
immich-server  |     2024-09-26T18:43:02.000Z,
immich-server  |     'IMAGE',
immich-server  |     false,
immich-server  |     false,
immich-server  |     '0:00:00.000000',
immich-server  |     true,
immich-server  |     'IMG20240926194302.jpg'
immich-server  |   ]
immich-server  | }

and the last logs I have are these

immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  |     at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
immich-server  |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
immich-server  |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:228:52)
immich-server  |     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
immich-server  |     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
immich-server  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich-server  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Object:
immich-server  | {
immich-server  |   "id": "31ac0f99-c2f8-4ebf-8158-864194552683",
immich-server  |   "source": "upload"
immich-server  | }
immich-server  | 
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  |     at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
immich-server  |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
immich-server  |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:228:52)
immich-server  |     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
immich-server  |     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
immich-server  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich-server  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich-server  | [Nest] 7  - 01/30/2025, 4:25:41 PM   ERROR [Microservices:JobService] Object:
immich-server  | {
immich-server  |   "id": "b29dcf4f-c1cb-4440-8794-16c6a179454a",
immich-server  |   "source": "upload"
immich-server  | }
immich-server  | 
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  |     at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
immich-server  |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
immich-server  |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:228:52)
immich-server  |     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
immich-server  |     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
immich-server  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich-server  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Object:
immich-server  | {
immich-server  |   "id": "aa0779b2-62c3-4e22-baee-ea29665ec9f2",
immich-server  |   "source": "upload"
immich-server  | }
immich-server  | 
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
immich-server  |     at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
immich-server  |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:47:26)
immich-server  |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:228:52)
immich-server  |     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
immich-server  |     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
immich-server  |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich-server  |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich-server  | [Nest] 7  - 01/30/2025, 4:25:48 PM   ERROR [Microservices:JobService] Object:
immich-server  | {
immich-server  |   "id": "148f3d7d-12da-4792-bd84-b3f2782e85ad",
immich-server  |   "source": "upload"
immich-server  | }
immich-server  |

These last ones I have no idea if they are caused by the issue or are the cause of the issue.

5 replies

Mraedis Jan 31, 2025
Collaborator

I don't really see much special here, other than the ML failing. If you do docker ps -a what is the uptime of everything? What's the RAM usage during these spikes?

The-Inamati Jan 31, 2025
Author

RAM increases but not to crazy amounts. I just finish some tests but were unfortunately inconclusive. I deployed from scratch a few versions between 1.112.1 and 1.125.7. All showed the same issue. I tried with storage template on and off with no big difference. Also tested in a "fresh" VM with nothing else running and also the same.

I ended up having to increases the number of cores to 12.

It still peaks but at least it doesn't slow down or crash the VM.

Any idea how I can go about and troubleshoot this?

Mraedis Feb 1, 2025
Collaborator

Well I'm just guessing but could it be your hardware not liking the load, and adding the cores spreads it a bit? I'm not too sure what you could do to test this other than make sure your concurrency settings aren't out of whack or limiting the docker container's CPU resources.

Aevaris Feb 1, 2025

probably a couple of silly questions but:

Is the host overprovisioned?
Does the CPU peg or only when running jobs/searches? (your graphic is only a 5min slice which is why I ask)

The-Inamati Feb 2, 2025
Author

It's not overprovisioned and it only happens when files are being uploaded to Immich. Other than that everything is fine.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Very High CPU Usage in Immich Server #15793

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Very High CPU Usage in Immich Server #15793

The-Inamati Jan 30, 2025

Replies: 2 comments · 5 replies

Mraedis Jan 30, 2025 Collaborator

The-Inamati Jan 30, 2025 Author

Mraedis Jan 31, 2025 Collaborator

The-Inamati Jan 31, 2025 Author

Mraedis Feb 1, 2025 Collaborator

Aevaris Feb 1, 2025

The-Inamati Feb 2, 2025 Author

The-Inamati
Jan 30, 2025

Replies: 2 comments 5 replies

Mraedis
Jan 30, 2025
Collaborator

The-Inamati
Jan 30, 2025
Author

Mraedis Jan 31, 2025
Collaborator

The-Inamati Jan 31, 2025
Author

Mraedis Feb 1, 2025
Collaborator

The-Inamati Feb 2, 2025
Author