Unknown error in logs when generating missing thumbnails from admin console

[stevehoek]

The bug

I've been noticing this error alot in the logs lately, when I go to generate missing thumbnails via the Jobs panel of the admin console (web)

2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Unknown error
2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Error: Unknown error
2025-01-26 19:05:11     at Sharp.toBuffer (/usr/src/app/node_modules/sharp/lib/output.js:163:17)
2025-01-26 19:05:11     at MediaRepository.decodeImage (/usr/src/app/dist/repositories/media.repository.js:54:68)
2025-01-26 19:05:11     at MediaService.generateImageThumbnails (/usr/src/app/dist/services/media.service.js:164:63)
2025-01-26 19:05:11     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-01-26 19:05:11     at async MediaService.handleGenerateThumbnails (/usr/src/app/dist/services/media.service.js:114:25)
2025-01-26 19:05:11     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
2025-01-26 19:05:11     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
2025-01-26 19:05:11     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2025-01-26 19:05:11     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Object:
2025-01-26 19:05:11 {
2025-01-26 19:05:11   "id": "bf1cbcd9-82fd-4e4f-ac4b-c482824f4fe8"
2025-01-26 19:05:11 }

The OS that Immich Server is running on

macOS Docker

Version of Immich Server

latest as of the date of this issue creation

Version of Immich Mobile App

latest as of the date of this issue creation

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

not relevant

Your .env content

not relevant

Reproduction steps

1. Log in as admin
2. Go to Admin console
3. Go to Jobs panel
4. Click Missing on the Thumbnail Generator job

Relevant log output

2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Unknown error
2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Error: Unknown error
2025-01-26 19:05:11     at Sharp.toBuffer (/usr/src/app/node_modules/sharp/lib/output.js:163:17)
2025-01-26 19:05:11     at MediaRepository.decodeImage (/usr/src/app/dist/repositories/media.repository.js:54:68)
2025-01-26 19:05:11     at MediaService.generateImageThumbnails (/usr/src/app/dist/services/media.service.js:164:63)
2025-01-26 19:05:11     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-01-26 19:05:11     at async MediaService.handleGenerateThumbnails (/usr/src/app/dist/services/media.service.js:114:25)
2025-01-26 19:05:11     at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:148:28)
2025-01-26 19:05:11     at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:130:13)
2025-01-26 19:05:11     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2025-01-26 19:05:11     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2025-01-26 19:05:11 [Nest] 7  - 01/26/2025, 7:05:11 PM   ERROR [Microservices:JobService] Object:
2025-01-26 19:05:11 {
2025-01-26 19:05:11   "id": "bf1cbcd9-82fd-4e4f-ac4b-c482824f4fe8"
2025-01-26 19:05:11 }

Additional information

No response

[theovds]

I might are seeing the same behavior (version 1.125.3). This was not the case with version 1.124.
If I run the missing thumbnail job, I see more than 1000 items that needs to be processed. Then, almost at the end, there is sometimes a crash and the Immich container starts again. If I try to run the job again, it starts again with the same nummer of thumbnails. I don't see anything related in the log (level: debug).

Edit: I went back to 1.124.2 and the missing thumbnail job is running normally (nothing to proceed).

[bo0tzz]

Can you set the log level to verbose and see if it logs any more detail?

[ssilence5]

Came here to report this same issue. Every night I get a warning of High CPU Usage and everytime it is trying to regenerate thumbnails for about 12,000 images. After it finished I tell it to regenerate for missing and it is always the same count. I will try to revert to an earlier version to see what happens.

[alextran1502]

@mertalev Do you have some input on the query we use to determine the missing thumbnail?

[HearthisAt]

Same here. Job is generating 48k+ Thumbnails on each missing run.

[alextran1502]

@HearthisAt Can you help me understand how you can identify the 48,000 thumbnails that were generated?

[HearthisAt]

Here are the logs and screenshot when I use on "missing" feature of the thumbnails job

immich.txt

Does this help?

[HearthisAt]

But: after a short visual check: most of the thumbnails and images are available. (99.x%)
More than 100k assets are within Jimmich.

[ssilence5]

This is the same thing I am seeing. Everytime I click missing thumbnails, there are over 12K that say they need to be generated. After finishing, if I click it again it says the same thing. It doesn't ever seem to finish. Moreover, this runs automatically every night spiking cpu usage.

[stevehoek]

Sometimes I find if I go to the photo ID that is generating the type of log message shown in the OP, I can choose to regen thumbnail and it works. Sometimes it doesnt. Could be some bad images that display fine in other software but can't gen a thumbnail?

[mertalev]

Is the number of jobs queued consistent with the number of people? There's a bug that queues all people to have their thumbnails generated (1 per person).

[ssilence5]

It does look to be if I look at all people including those hidden.

[stevehoek]

In my case it is close to that number but I also think I have some images that fail thumbnail generation... is there no explicit log entry for when that happens?

Really looking forward to the rewritten Repair feature as I have a big library that I suspect has some orphans and such

[HearthisAt]

Hmm..
One account has 10.800 people including hidden with ML setting minimum 1 image.
1.047 another account.

Hard to believe that this will sum up to 40k.

But wait: Are Thumbs & Previews of one asset are counted as 2 Thumbnails in generation?

[matticas]

I am getting 130k thumbnail generation, on account of about 130k photos, mostly in external library.

People / faces is around 25k.

[HearthisAt]

External RO Library with nearly 11k assets.

[mertalev]

I also think I have some images that fail thumbnail generation... is there no explicit log entry for when that happens?

Failures do get logged with the error and stack trace. Unfortunately we can only report what the image library itself reports, so if it says "unknown error" then there's not much to work with.

Hmm..
One account has 10.800 people including hidden with ML setting minimum 1 image.
1.047 another account.

Are those the only two accounts? Can you check the person table itself to see how many rows it has?

But wait: Are Thumbs & Previews of one asset are counted as 2 Thumbnails in generation?

No, the thumbnail and preview is generated in the same job. Person thumbnail jobs also only generate 1 file; there's no preview for them.

[HearthisAt]

Can you please send the commands for postgres to get these numbers?

[mertalev]

docker exec immich_postgres psql --dbname=immich --username=postgres -c "select count(*) from person"

[HearthisAt]

Thanks. It is exactly that amount of thumbnails: 40182

[mertalev]

Sweet! That matches the behavior I see locally and is fixed with #15734