Immich is keeping HDD running

[zenkraker]

The bug

I use Truenas scale 24.10.
I passed a pool to immich to be used as main entryPoint for my data.
All other volumes are running on SSDs, even postgres DB all is running on another ZFS pool made for applications.
The problem is that even though no data are accessed for a long time (even for days), I mean, I did not access immich UI and/or app, no upload occurred etc. the underlying disks of the ZFS pool never spin down.
This ZFS pool is exclusively used by Immich and there no other application in my server that uses this pool.

To experiment I just stopped the docker compose with immich and the disks spinned down as expected.

The OS that Immich Server is running on

Debian

Version of Immich Server

v0.121.0

Version of Immich Mobile App

v0.121.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  immich-server:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    container_name: immich_server
    # user: ${PUID}:${PGID}
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
      NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES}
      NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES}
      IMMICH_WORKERS_INCLUDE: "api"
      IMMICH_CONFIG_FILE: /config/immich-config.yaml
    env_file:
      - stack.env
    runtime: nvidia
    labels:
      traefik.enable: true
      traefik.http.routers.immich-intern.rule: Host(`immich.home.${DOMAIN}`)
      traefik.http.routers.immich-intern.entrypoints: https
      traefik.http.routers.immich-intern.service: immich-service
      traefik.http.routers.immich-extern.rule: Host(`immich.${DOMAIN}`)
      traefik.http.routers.immich-extern.entrypoints: https-external
      traefik.http.routers.immich-extern.service: immich-service
      traefik.http.services.immich-service.loadbalancer.server.port: 2283
      traefik.docker.network: proxy
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload:rw #This is the path where all files you upload with cli go
      - ${BASE_PATH}/immich-server:/config
      - ${BASE_CONFIG_PATH}/immich/immich_config/immich-config.yaml:/config/immich-config.yaml
      #- /mnt/remotes/TRUENAS_Photos/Immich/external_libraries/:/external_libraries NO need to set it if you don't use external libraries
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    networks:
      proxy:
      immich_internal:

  immich-microservice:
    container_name: immich_microservice
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # user: ${PUID}:${PGID}
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
      NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES}
      NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES}
      IMMICH_WORKERS_EXCLUDE: "api"
    env_file:
      - stack.env
    runtime: nvidia
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload:rw #This is the path where all files you upload with cli go
      - ${BASE_PATH}/immich-server:/config
      #- /mnt/remotes/TRUENAS_Photos/Immich/external_libraries/:/external_libraries NO need to set it if you don't use external libraries
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - redis
      - database
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    networks:
      proxy:
      immich_internal:

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    # user: ${PUID}:${PGID}
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
      NVIDIA_VISIBLE_DEVICES: ${NVIDIA_GPU}
      NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES}
    env_file:
      - stack.env
    networks:
      immich_internal:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    volumes:
      - ${BASE_PATH}/immich-server/machine-learning:/cache

  redis:
    container_name: immich_redis
    image: ${REDIS_IMAGE}
    # user: ${PUID}:${PGID}
    restart: unless-stopped
    networks:
      immich_internal:
    healthcheck:
      test: redis-cli ping || exit 1

  database:
    container_name: immich_postgres
    image: ${POSTGRES_IMAGE}
    # user: ${PUID}:${PGID}
    restart: unless-stopped
    networks:
      immich_internal:
    environment:
      TZ: Europe/Berlin
      PUID: ${PUID}
      PGID: ${PGID}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: "--data-checksums"
    env_file:
      - stack.env
    volumes:
      - ${BASE_PATH}/immich-postgres:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        "postgres",
        "-c",
        "shared_preload_libraries=vectors.so",
        "-c",
        'search_path="$$user", public, vectors',
        "-c",
        "logging_collector=on",
        "-c",
        "max_wal_size=2GB",
        "-c",
        "shared_buffers=512MB",
        "-c",
        "wal_compression=on",
      ]

networks:
  proxy:
    external: true
  immich_internal:
    name: immich_internal
## there is a known issue with Traefik: see here https://github.com/immich-app/immich/discussions/437#discussioncomment-3609797

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/ImportantPool/Photos_Immich/

NVIDIA_DRIVER_CAPABILITIES=all # If using NVIDIA GPU     

#IMMICH_CONFIG_FILE=/usr/src/app/config
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v1.121.0
IMMICH_LOG_LEVEL=verbose

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_PASSWORD=blablabla

REDIS_HOSTNAME=redis
REDIS_PORT=6379

REDIS_IMAGE=redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8
POSTGRES_IMAGE=tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0

Reproduction steps

1. Use a ZFS pool that is used only by immich
2. Let immich run and the disks of the pool will never spin down
3. Turn off immich compose, wait for spin down in truenas scale (must be applied to the disks of the pool)
4. Disks should spin down right after the setting of spin down in truenas

Relevant log output

No response

Additional information

No response

[mmomjian]

You can disable the docker health checks if you prefer to not have any disk activity. That might help.

[zenkraker]

HI @mmomjian thanks for the answer :)

I've updated the post, because I noticed that I did not properly mention that the volumes of database and other "config" folders for immich are all running on a separated ZFS (SSDs) pool.

So to recap:

1. HDDs pool is used only for /usr/src/app/upload
2. The rest of volumes (postgres as well) are on a SSD pool

I can try to disable healthChecks for docker, but I'd guess that situation won't improve since the healthCheck are for postgres only and it uses only a SSD pool.

I keep you updated

[mmomjian]

Every container has a healthcheck. You can look at our current docker compose for an example of how to disable (change false to true)

[zenkraker]

@mmomjian I just tested and disabled healthCheck as you suggested and you were right! This stops the disks from keep running. Why does this happen though? Do you check in the healthCheck that the underlying disk is available/online?

[mmomjian]

This is the expected behavior. The health check should confirm that the underlying storage is behaving properly.

[mertalev]

The server and machine learning health checks don't check the disk. I imagine it's the Postgres health check that causes disk access, but this health check is incidentally not useful for ZFS since ZFS already does checksumming.