[Feature] Outsource video transcoding

[sebbi08]

The feature

It would be nice to have an option to "outsource" video transcoding.

My immich instance is on a small Intel NUC and transcoding my 4k footage is super slow. So what would be nice is if I could use by beefy gaming PC to transcode the videos. So after some vacation I can turn an extra "server" on my pc to transcode for some hours and shut it down again, and the main server would not do any encoding at all if the server is not available.

Platform

• Server
• Web
• Mobile

[bo0tzz]

You can run multiple instances of the microservices container and put those wherever you like, such as on your pc. Right now any instance will pick up any jobs, but in the future we hope to add an option to filter which jobs will be accepted through an env var.

[aisbergde]

That's an interesting idea: to use different instances on different devices. Does it mean, an instance will grab a "task", the task is marked to be currently processed to avoid parallel processing by other instances?

It would be useful to mention this option for parallel processing in the docs. Especially when adding new and big external libraries, it takes days to process them. External help could accelerate this.

[sebbi08]

But wouldn't this need the dedicated service to be connected to the original DB and file system ? And if yes you know a "simple" way of doing this ?

[aisbergde]

I don't know a simple way and how to set up. That's why I ask to document, how to do.

[AngelaDMerkel]

This is not all the difficult to do with a docker swarm and you could configure the networking rather seamlessly

[rasoga]

You can run multiple instances of the microservices container and put those wherever you like, such as on your pc. Right now any instance will pick up any jobs, but in the future we hope to add an option to filter which jobs will be accepted through an env var.

This is good to know. Where do I then setup concurrency? There is a Setting for e.g. thumbnail-generation or transcoding. Let's say I deploy 3 microservices across 3 machines. Do they each pick up the respective number of tasks (per default 5 for thumbnails and 1 for transcoding) or do I need to increase the concurrency-setting myself?

[AlmightyFrog]

You can run multiple instances of the microservices container and put those wherever you like, such as on your pc. Right now any instance will pick up any jobs, but in the future we hope to add an option to filter which jobs will be accepted through an env var.

How can i specify the host to check? For machine learing i can change Machine Learning Settings > URL > "http://immich-machine-learning:3003" which works perfectly fine for offloading to other machine in my network.
But i can't find anything similar for microservices. Woud you kindly hint me where to look @bo0tzz?

I've already mounted all my library files necessary and database is also reachable. On my other host container is already running, i just can't make my main host to use it.

---

Update 1

I found an semi elegant way to modify the hosts file using docker compose file only

extra_hosts:
  - "immich-microservices:192.168.123.231"

but it seems there is a harder binding between immich-server (?) scheduling immich-microservices.
In case i renamed the immich-microservices container to something else - it still found it.
In case i removed the immich-microservices container, nothing happened anymore for e.g. thumbnail creation.
I checked in the running immich-server itself with curl, that immich-microservices:3002 was resolved to correct host 192.168.123.231 and got a correct html response.

---

Update 2

Really strange behavior. Even without using any tricks like extra_hosts there seems to be a strong binding.
Can someone help me explain how immich-server triggers immich-microservices?

In docker compose i even tried clean adding nginx as loadbalancer mimic the container name

immich-microservices:
    container_name: immich_microservices
    image: nginx
    volumes:
      - ./nginx/:/etc/nginx/conf.d/

but still same behavior that either a) nothing works or b) the whatever-named-original-microservices container is uses whichever strange name i give it.

[mertalev]

1. Yes.
2. Yes.
3. Job scheduling is handled in microservices, which queries the ML service as part of its jobs. As a result, there is no need to connect the ML service to Redis. This also makes it easier to set up remote machine learning.

[AlmightyFrog]

Cool, thank you very much for your answers. I was not sure, but it seems to be really fine. The only issue i have when doing microservices on multiple hosts is, that it seems like the rules seem to be global:

My main machine is a small 4c/4t machine, barely able to do two parallel VP9 transcodings. The other machine i have for testing is a 6c/12t which can happily execute 5 parallel VP9 transcodings.
I've set now the number to 4, so that both of them take up to 4 jobs which seems to be a good compromise.

Also, transcoding settings are for both, so i can't mix the encoding like software/quicksync/nvenc sadly. Best I can do is to bind all to sw encoding which is a little sad.
Or am I missing something?

[mertalev]

Transcoding falls back to software if hardware encoding fails, so you can still have it enabled. You're right though that the workload will be split evenly between the two even though one is more capable than the other. In the future, we (and by we I mean @zackpollard :D) intend to change this so instances can have unequal workloads.

[zackpollard]

Yea, as @mertalev says, I have plans to vastly improve the existing worker setup and part of that will hopefully be enabling/disabling certain functionality on certain workers, changing concurrency per worker and overriding default settings per worker too (so you could have one machine using quicksync hw acceleration and another using nvidia for example). Not sure where this sits in my order of priorities, but I want to be able to do exactly this for my personal setup, so it's not at the bottom at least 😄

[AlmightyFrog]

That sounds cool.
@mertalev the issue with using the fallback on sw is that at least how i remember it e.g. using nvenc it did repeatedly load stuff in video ram, see it is failing removing it and falling back to sw over and over again. Not sure but i guess that takes also some time.

What would be good would be a kind of capabilities test you can run within the immich container once. After that, you by heart know what really works and what not (e.g. i can't get my intel 7th gen and 10th gen quicksync vp9 encoding to get running). Then you could with or without @zackpollard planned adaptiations make a kind of automatic overriding settings per client chose there the preset which is really working.

---

Update: I am pretty happy. It really is that simple to scale now with just spin up a new docker container somewhere, really nice. But letting it run also on my desktop (2c/4t) with same settings have 4 parallel VP9 transcodings show the issue of assymetric powerfull systems quite good.

[Cakeday]

I have the microservices container offloaded to a different host:

My heavy lifter (MacBook):

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    restart: always
   

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    ports:
      - 3003:3003


volumes:
  model-cache:

on the heavy lifter:
.env

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/Volumes/mount

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=**********

DB_HOSTNAME=LOW_POWER_CLIENT_IP # replace with theLAN ip of the machine hosting your Postgres/redis, same below
REDIS_HOSTNAME=LOW_POWER_CLIENT_IP

TZ=***********

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=********
DB_DATABASE_NAME=********

Notice the exposed ports on my lower-powered host machine

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        limits:
          # Number of CPU threads
          cpus: '3.00'
          # Gigabytes of memory
          memory: '1.5gb'

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always
    ports:
      - 6379:6379

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
    ports:
      - 5432:5432

volumes:
  model-cache:

The NFS share settings I'm using to share to my MacBook over LAN:

/home/cakedayy/docker_apps/immich/mount <my_client_ip>(rw,sync,no_root_squash,no_subtree_check,insecure)

[Qhilm]

Thanks a lot for this. What is the impact to Immich when your workhorse/macbook is offline? I have already offloaded the machine learning container to my macbook, but I'm not sure I fully understand all the tasks the microservices container is responsible for.

[surajd27]

This works well for me thanks @Cakeday for the comprehensive guide.

@Qhilm When my more powerful remote machine is offline, the jobs just get queued on the host, and as soon as it comes online, it pick up the jobs. Occasionally, you need to restart the services on the remote machine, but normally works seamlessly.

I did have to make a few changes to the above, however, because I had issues with the remote machine connecting:

First change:
DB_HOSTNAME='192.168.1.252'
REDIS_HOSTNAME='192.168.1.252'

This wouldn't connect even if I added the ports to the IPs above. What I found is that you have to keep the above as just IP address of the host on the local network and add the ports so it looks like this:

DB_HOSTNAME='192.168.1.252'
DB_PORT:'5432'
REDIS_HOSTNAME='192.168.1.252'
REDIS_PORT='6739'

Second change/addition:
The network drives where upload etc. is does not mount in my case. To fix this, I added the following to the remote machine docker-compose.yml to add the network share as a cifs mount:

[In the immich-microservices section]
volumes:
- upload:/usr/src/app/upload

volumes:
upload:
driver: local
driver_opts:
type: cifs
o: "username=,password=,rw,uid=,gid="
device: "//ds124/docker/immich-app/library"

Replace , , and . To get the userid and groupid, you ssh to your host and run the id command. You can add as many of these directories e.g. for external libraries as needed - just remember to add them in both places.

[Qhilm]

Immich recently added the ability to have multiple machine learning URLs, which it will try for each picture in sequence. (it's a prioritised list, basically) That is actually sufficient for me. When I see my poor NAS dying, I just fire up the machine learning on my Mac and it takes the load away. But if my mac is powered off or I'm not home, the machine learning doesn't just stop, hence I prefer it I think.

[erkexzcx]

For those coming via Google, I wrote guide here: #14142