Dedup does not work with imagestore - uses wrong path for image layers

[ngopalak-redhat]

The Dedup() function in the overlay driver does not correctly handle layers when an imagestore is configured. It always uses the main graphRoot path instead of the imagestore path, preventing deduplication from working on image layers stored in a separate imagestore.

In storage/drivers/overlay/overlay.go line 2810, the Dedup() function calls:
dir, _, inAdditionalStore := d.dir2(layer, false)

When imagestore is configured (e.g., /var/lib/containers/imagestore), image layers are stored at /var/lib/containers/imagestore/overlay/<layer-id>/diff. The Dedup() function should operate on these imagestore paths.

cc: @haircommander @asahay19

[mtrmac]

In storage/drivers/overlay/overlay.go line 2810, the Dedup() function calls: dir, _, inAdditionalStore := d.dir2(layer, false)

When imagestore is configured (e.g., /var/lib/containers/imagestore), image layers are stored at /var/lib/containers/imagestore/overlay/<layer-id>/diff.

… and dir2 finds that, AFAICS. If dir2 couldn’t find a layer in an image store, I’d expect we would hear much more urgent reports about imagestore not working at all.

---

The difference is, I think, the inAdditionalStore return value of dir2, with the code explicitly choosing not to deduplicate in non-primary stores.

It’s somewhat unclear whether the image store should be in scope. On the one hand, “the driver writes to both and maintains both” means that deduplication might be expected to happen, there is no-one else to do that. But then again, the whole purpose of imagestore is to use different filesystems with different characteristics for the image layers and container layers, so I don’t think deduplicating across them would be ever expected to work.

But it’s a significant complication in light of #961 : if deduplication should be expected to work across two different directories (with two different mounts and probably two different filesystems), the whole “detect in advance whether reflinks can work” question gets much more messy.

[ngopalak-redhat]

@mtrmac Here are the recreate steps:

1. Create Ubuntu VM on GCP - Uses ext4 FS which doesn't support reflinks
2. Setup local-up-cluster with cri-o
3. Modify cri-o https://github.com/cri-o/cri-o/blob/main/server/image_pull.go#L199 to invoke:

store.Dedup(cstorage.DedupArgs{
		Options: cstorage.DedupOptions{
			HashMethod: cstorage.DedupHashSHA256,
		},
	})

4. Add the imagestore config:
   Create a xfs disk and attach to GCP ubuntu instanace. Then:

cat 02-imagestore.conf
[crio]
  # Use XFS disk for image storage (read-only image layers)
  imagestore = "/var/lib/containers/imagestore"

5. Add some log to know if the dedup happens:

savedMB := float64(result.Deduped) / (1024 * 1024)
log.Infof(ctx, "Deduplication complete for image %s: saved %.2f MB in %v",
		imageID, savedMB, duration)

6. Pull few images where cross image dedup is possible:

  kubectl run postgres-alpine --image=docker.io/library/postgres:16-alpine --env="POSTGRES_PASSWORD=test"

  kubectl run python-alpine --image=docker.io/library/python:3.12-alpine -- sleep 3600

  kubectl run node-alpine --image=docker.io/library/node:20-alpine -- sleep 3600

  kubectl run ruby-alpine --image=docker.io/library/ruby:3.3-alpine -- sleep 3600

*** Failure Log ***

journalctl -u crio --since "200 seconds ago" -f | grep -E "saved.*MB"
Jul 10 01:28:56 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[308615]: time="2026-07-10T01:28:56.592396421Z" level=info msg="Deduplication complete for image 2893cf9f584f34cfa2d0b8dd6ba9e63dfacd1e1105a304bb4f72317b00f89029: saved 0.00 MB in 93.946µs" id=0a754001-b6af-4e70-8ca2-0be8a336abc9 name=/runtime.v1.ImageService/PullImage
Jul 10 01:29:03 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[308615]: time="2026-07-10T01:29:03.331284347Z" level=info msg="Deduplication complete for image de3a4eab8fdfa507ea92aac488b916b08089e515db49b055fe71dfa271ba3a28: saved 0.00 MB in 148.701µs" id=6370541a-3804-4f31-84b4-01135dc6591d name=/runtime.v1.ImageService/PullImage
Jul 10 01:29:07 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[308615]: time="2026-07-10T01:29:07.968334022Z" level=info msg="Deduplication complete for image 11cedc39e663e7c5d5cb9cc77a461a0d2adc25537b94e6831a6108f09cb2001b: saved 0.00 MB in 155.047µs" id=ddf519e0-1d18-4551-a93a-d1e3b4d631fc name=/runtime.v1.ImageService/PullImage
Jul 10 01:29:11 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[308615]: time="2026-07-10T01:29:11.494753967Z" level=info msg="Deduplication complete for image 96142f41350918e8a043a93a2e4a0fafa0cc82fa07983eb515a12f3bc6353696: saved 0.00 MB in 156.823µs" id=a5c29f62-26cd-40d8-86ed-d40a38f7b55e name=/runtime.v1.ImageService/PullImage

You can see 0 MB saved. Here 0MB does not mean that ref-link is not supported. Here it means that imageStore is not supported.

Add the fix:

func (d *Driver) Dedup(req graphdriver.DedupArgs) (graphdriver.DedupResult, error) {
	var dirs []string
	for _, layer := range req.Layers {
		dir, _, inAdditionalStore := d.dir2(layer, false) - Remove
		dir, _, inAdditionalStore := d.dir2(layer, true) - Add

*** Get the log again - Its a success ***

journalctl -u crio --since "10 seconds ago" -f | grep -E "saved.*MB"
Jul 10 01:54:02 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[335749]: time="2026-07-10T01:54:02.731113162Z" level=info msg="Deduplication complete for image de3a4eab8fdfa507ea92aac488b916b08089e515db49b055fe71dfa271ba3a28: saved 0.67 MB in 819.071329ms" id=2fdcb412-d8eb-447f-a12f-95463a21332e name=/runtime.v1.ImageService/PullImage
Jul 10 01:54:07 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[335749]: time="2026-07-10T01:54:07.049477718Z" level=info msg="Deduplication complete for image 2893cf9f584f34cfa2d0b8dd6ba9e63dfacd1e1105a304bb4f72317b00f89029: saved 4.09 MB in 935.497083ms" id=56ebc1b8-36f4-497a-9f9c-c2688a392c8d name=/runtime.v1.ImageService/PullImage
Jul 10 01:54:12 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[335749]: time="2026-07-10T01:54:12.979895296Z" level=info msg="Deduplication complete for image 11cedc39e663e7c5d5cb9cc77a461a0d2adc25537b94e6831a6108f09cb2001b: saved 6.24 MB in 1.258883377s" id=5eddd6b1-72c0-4d29-bf90-13a5a79f6b19 name=/runtime.v1.ImageService/PullImage
Jul 10 01:54:17 ngopalak-ubuntu.asia-south1-b.c.openshift-gce-devel.internal crio[335749]: time="2026-07-10T01:54:17.846963164Z" level=info msg="Deduplication complete for image 96142f41350918e8a043a93a2e4a0fafa0cc82fa07983eb515a12f3bc6353696: saved 6.62 MB in 1.362813477s" id=9495911c-5871-4400-81c8-e27915e29325 name=/runtime.v1.ImageService/PullImage

This is not a problem in RHEL as the boot disk itself is XFS. So most customer won't be impacted. I was trying to build a automated test case to measure the impact of invoking Dedup on every image pull. The results are looking good as its only a few seconds it takes to Dedup (see above output). I'm yet to test incremental Dedup with a very large number of images.

If imageStore is not supported I have to work on bringing up my test env on RHEL.

[mtrmac]

Add the fix:

func (d *Driver) Dedup(req graphdriver.DedupArgs) (graphdriver.DedupResult, error) {
	var dirs []string
	for _, layer := range req.Layers {
		dir, _, inAdditionalStore := d.dir2(layer, false) - Remove
		dir, _, inAdditionalStore := d.dir2(layer, true) - Add

I’m sure this does something (compare the analysis in #980 (comment) ) but it just can’t be the right fix; it would completely ignore the container layers in d.home.