[Feature] Avoid duplicating assets and save server resources and storage space

[dpantel]

The feature

Media assets should be stored on a per-instance basis, rather than a per-user basis. This should be true for both uploaded media, as well as generated media.

Let's say I take a 20-minute video with my phone. Then I share it with my family members, who save it to their phones, and who all have the Immich app installed. Now Immich will upload multiple copies of the same video to the server, and then re-encode this video multiple times, once for each user.

Let's also say I have an External Library directory with 5000 photos of a family trip. I want to add that directory to each Immich account, so everyone can see those photos. Immich will now generate and store 5000 thumbnails for each user.

Basically, there is a lot of resource waste, both in storage, and in generating encoded video, thumbnails, etc. I think a lot can be saved by checking if an asset (uploaded or generated) already exists somewhere on the server before uploading/generating.

I think this can be accomplished relatively easily, either by co-mingling storage into one single pool for all users, or keeping assets nominally separate, but using hard-linking.

Platform

• Server
• Web
• Mobile

[v411e]

I would highly appreciate this feature. My use case is like this:

• I want to give my family members access to certain external libraries as we (obviously) share some common history like family vacations etc.
• As you pointed out, having all those Previews generated and stored multiple times on a per-user basis does waste a lot of resources and makes the whole multi-user thing a bit useless. I could also just create a dedicated Immich Instance per user and have a similiar resource footprint (at least storage wise)

[TheMarf]

This would be a great feature. I had just commented on the immich-go repo asking about reducing duplicates because my wife and I have shared so many pictures with each other and there are so many duplicates of the same image even though they are attached to different user keys. It sounds like this is likely something that needs to be addressed on the immich side.

[blublub]

I was looking to achieve that via a zfs partition in my Immich VM. Problem is that the copy in write feature of ZFS ruins that.
I have 1.8TB of image data and when I copy that onto a ZFS partition that vhdx has the size of 1.9TB and zpool shows a usage of 1TB - so 0.8TB could have been saved but got wasted in the process - so not worth it....

[Crocmagnon]

I'm looking for a photo backup solution that has this kind of feature, specifically with the iCloud Shared Photo Library usecase in mind. I'm currently uploading everything to nextcloud, but it doesn't manage duplicates either.