[Feature] Automatically pause and resume Remote Machine Learning for on-and-off beefy machine

[jucor]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

I love how Immich allows to offload machine learning to a remote computer, especially useful when running Immich on a low-CPU NAS.

Would it be possible to automatically "pause" the ML queue when the remote ML computer is offline, and automatically resume the processing when the remote computer is back online?

Use case: Immich on a NAS, and remote ML computer being my beefy laptop (MacBook Pro M1 Max or Ultra). My laptop is sometimes offline, e.g. because I'm not home. If I understand the documentation correctly, this would lead to jobs crashing that I would have to manually resume. Pausing the queue instead of crashing, and polling regularly then resuming, would be terrific.

Thanks for considering!

Platform

• Server
• Web
• Mobile

[filski]

To add a slight variation too - it would be useful to be able to specify primary/backup ML instances. I.e. if the external machine is available use that, otherwise use the local ML container. That way could spin up docker/faster laptop for ML when doing a large import, but for general phone backups etc it would keep using the local ML container. The workaround is to switch the ML URL manually, so not onerous.

[Zedarius5]

Yes, this. I just spent an hour trying to figure out an nginx load balancer, but had trouble getting it to talk properly to the docker container. Would love auto-fallback onto the local ML if the external ML machine isn't reachable.

Should we submit this as a standalone feature request? I'm new to github, so I'm not sure the proper channels.

[mertalev]

Rather than pausing the queue, I think it'd be more elegant to retry failed jobs. Right now, a failed job means the admin has to manually requeue it, but it'd be better if the job scheduling were more fault tolerant. That would also help with more transient errors like SocketError, etc.

[jucor]

True, but it would mean the immich server would keep trying and failing and trying and failing in a loooong loop until the remote ML service is back on.

[mertalev]

It could probably do both, intermittently checking if the server is still up and pause/resume based on that. Retries would handle the jobs that were running when this happens along with more transient errors.