Image has different digests in Docker and Podman

[sshnaidm]

Issue Description

When pulling same image from same registry with docker and podman, they have a different digest. The docker diges is same as in registry.

$ curl -I -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json"      -H "Authorization: Bearer $TOKEN"      "https://registry-1.docker.io/v2/library/alpine/manifests/latest"  | grep sha256
docker-content-digest: sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
etag: "sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c"

$ docker inspect  docker.io/library/alpine:latest | grep -i digest -A2                                                                                                                       
        "RepoDigests": [
            "alpine@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c"  # same as in registry
        ],                                                                                               
$ podman inspect docker.io/library/alpine:latest | grep -i digest -A2      
          "Digest": "sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474",
          "RepoTags": [                                                                                  
               "docker.io/library/alpine:latest"
--                                         
          "RepoDigests": [  
               "docker.io/library/alpine@sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474",  # what is this
               "docker.io/library/alpine@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c"  # same as in registry

Podman has two "RepoDigests" and in "Digest" it has a different one from Docker.

Steps to reproduce the issue

Steps to reproduce the issue

1. docker pull docker.io/library/alpine:latest
2. podman pull docker.io/library/alpine:latest
3. inspect images and compare digests

Describe the results you received

Podman has two digests in "RepoDigests" and one of them is as in Docker, but in "Digest" field it has a different sha256

Describe the results you expected

Docker and Podman digests are same

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 90.76
    systemPercent: 2.72
    userPercent: 6.52
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "41"
  eventLogger: journald
  freeLocks: 901
  hostname: sshnaidm.remote
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.12.11-200.fc41.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2070339584
  memTotal: 67158261760
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250121.g4f2c8e7-2.fc41.x86_64
    version: |
      pasta 0^20250121.g4f2c8e7-2.fc41.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 651235328
  swapTotal: 8589930496
  uptime: 92h 24m 32.00s (Approximately 3.83 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/sshnaidm/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/sshnaidm/.local/share/containers/storage
  graphRootAllocated: 502921060352
  graphRootUsed: 435261177856
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 810
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /storage/containers/storage/volumes
version:
  APIVersion: 5.3.2
  Built: 1737504000
  BuiltTime: Wed Jan 22 02:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

There is #3761 but not sure if that is the same thing
cc @mtrmac

[mtrmac]

1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474 is the per-platform amd64 image; a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c is the top multi-platform index.

I think listing both in RepoDigests is correct — both are valid ways to refer to that image.

As for the top-level Digest field, well, no promises can structurally be made — images are deduplicated in local storage, so there may be many valid digests that are “the right source” for that image, but the Digest field can only hold one value. The code has some behavior here (it even depends on how you refer to that image!) but we are not, AFAIK, making any explicit promises that we are going to choose one of the possibly many values in any predictable way.

#3761 is relevant to the wider theme that RepoDigests is, sadly, not really reliable either.

[sshnaidm]

Well, in Docker there is only one in RepoDigests and it's similar to registry's one. I mean here more for Docker-Podman compatibility as well.
#3761 I think is not related, there is a "correct" digest here in RepoDigests, but it's not in Digest field and it's "one of". It makes problem with detection whether we need to pull the image or not, was it changed in the registry or same.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

Fundamentally the expectation that an automation can pull repo:tag and then expect the single reported Digest to match exactly one predicted value is untenable as a general feature with image deduplication.

Someone else could have written exactly the same automation for a differently-compressed (= different digest) of the same image, and both automations can’t be right about their prediction.

Or the admin might have set up a workflow to pre-load repo:tag, using a digest reference, and for a multi-platform image, chosen the unexpected one from the two possible digests. Again, both pulls can’t be right about their digest prediction.

(This could only ever work with very closed setups which ~never admit third-party images or customized product distribution workflows. But those kinds of systems probably already know when images have been updated, and don’t really need costly polling for tag changes.)