Recurring Layer Size Verification Error with Trivy Scanner in Harbor (Error: got 0, want [size])

[kaec-santos]

Hello Harbor team,

I'm experiencing a recurring issue when scanning images using Trivy in Harbor. The problem started after deleting a specific image layer, and since then, it has been affecting multiple applications.

🐞 Issue Details:

Error Message:
failed to analyze layer (sha256:303e23...): unable to get uncompressed layer sha256:303e23...: failed to get the layer content: error verifying size; got 0, want 28661744
HTTP Response: 500 status code when retrieving the vulnerability report.
Trivy Version: v0.56.1
Harbor Version: v2.0
Redis Version: v7.4.0

✅ What I’ve Checked:
The image builds without errors.
The image digest in the registry is correct.
Trivy’s vulnerability database is up-to-date.
Pulling the image manually (docker pull) results in the same error.

🔍 Troubleshooting Attempts:
Redis Cache: We suspect this could be related to Redis, as suggested in the Harbor community. I flushed the Redis DB (FLUSHDB command) as advised, but the issue persists.
Impact: After deleting a problematic layer, this issue has become frequent, affecting different versions of the same image.

❓ Questions:
Could this be related to data corruption in Redis or cache inconsistencies?
Is there an automated cleanup process for broken Redis entries in Harbor? We didn’t find logs indicating this process.
Are there specific logs or components in Harbor that we should investigate further?
Any guidance would be greatly appreciated. Let me know if more information is needed to assist with troubleshooting. 🙏

[Vad1mo]

we just had this issue in the slack channel

echo 'FLUSHDB' | redis-cli -h ${redis_master} -p 6379 -x -n 1

[Vad1mo]

I see you already did a cache reset, can you try to delete the affected images that have this layer in harbor and run GC, it should work than

[Vad1mo]

Also you should update Harbor to the latest version.

[kaec-santos]

@Vad1mo, sorry, but we are currently on version 2.12.0. The error is happening because the key with prefix blobs:: (and type hash) has a key "size" with value == 0

[Vad1mo]

So in your case, resting the cache didn't work right?

[stonezdj]

Maybe you have deleted the blob with digest sha256:303e23... from the file system. or it data file is corrupt. You should recover the data file: current its file size is 0.

/data/registry/docker/registry/v2/blobs/sha256/30/303e23../data

[reasonerjt]

@kaec-santos
You mentioned

The problem started after deleting a specific image layer ...

Did you delete the layer from storage? I believe you should NEVER do this b/c this essentially make the image unusable, it's not only impacting scanning but no client will be able to pull this image.