FreeBSD14.2 Podman 5.2.5_2 error on push to docker hub: error: writing blob: patch .. happened during read: digest did not match, expected ..

[dragospacher]

Issue Description

Podman 5.2.5 on FreeBSD 14.2, FreeBSD updated to 9feb2025, running as root:

root@freebsd:~ # podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/dragos35/freebsd14.2 latest 627ca8838133 38 minutes ago 514 MB
localhost/freebsd 14.2 627ca8838133 38 minutes ago 514 MB

root@freebsd:~ # podman push docker.io/dragos35/freebsd14.2:latest
Getting image source signatures
Copying blob ea5ddf9d8bc9 done |
Error: writing blob: Patch "https://registry-1.docker.io/v2/dragos35/freebsd14.2/blobs/uploads/220d7490-b231-4225-869e-3edb7004397c?_state=PnPhPReIIABiY967HdPdWf9GumOHh-tVhMWovIRiOSx7Ik5hbWUiOiJkcmFnb3MzNS9mcmVlYnNkMTQuMiIsIlVVSUQiOiIyMjBkNzQ5MC1iMjMxLTQyMjUtODY5ZS0zZWRiNzAwNDM5N2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMDItMDlUMDY6MzU6NDEuMjIyNjU0OTM5WiJ9": happened during read: Digest did not match, expected sha256:ea5ddf9d8bc93e31d35d234af37f14773cd78efdca835c6303de2556966e12e8, got sha256:01ae2daa9fbc8a7d8c34942393c8717273c20bcbc62455861c3d4de476e21bab

root@freebsd:~ # df -h | grep fdesc
fdescfs 1.0K 0B 1.0K 0% /dev/fd

Please let me know if you need more info to debug this.

Steps to reproduce the issue

Steps to reproduce the issue

1. freebsd-update fetch
2. freebsd-update install
3. pkg install podman
4. mount -t cd9660 /dev/md0 /mnt
5. cp /usr/local/etc/containers/pf.conf.sample /etc/pf.conf
6. vim /etc/pf.conf and replaced with my network card name
7. fetch https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/14.2/FreeBSD-14.2-RELEASE-amd64-memstick.img
8. mdconfig -a -t vnode -f FreeBSD-14.2-RELEASE-amd64-memstick.img -u 07. mkdir freebsd-rootfs8. tar -xJf /mnt/usr/freebsd-dist/base.txz -C freebsd-rootfs9. podman import freebsd-rootfs.tar freebsd:minimal
9. (podman images)
10. podman run -it freebsd:minimal /bin/sh0 # exit form container # we see its running well
11. podman commit $(podman ps -lq) my-freebsd:latest
12. podman tag localhost/my-freebsd:latest docker.io/dragos35/freebsd14.2:latest
13. podman push docker.io/dragos35/freebsd14.2:latest
14. error

Describe the results you received

Looks like SHA hash differences in some blob.

root@freebsd:~ # podman push docker.io/dragos35/freebsd14.2:latest
Getting image source signatures
Copying blob ea5ddf9d8bc9 done |
Error: writing blob: Patch "https://registry-1.docker.io/v2/dragos35/freebsd14.2/blobs/uploads/220d7490-b231-4225-869e-3edb7004397c?_state=PnPhPReIIABiY967HdPdWf9GumOHh-tVhMWovIRiOSx7Ik5hbWUiOiJkcmFnb3MzNS9mcmVlYnNkMTQuMiIsIlVVSUQiOiIyMjBkNzQ5MC1iMjMxLTQyMjUtODY5ZS0zZWRiNzAwNDM5N2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMDItMDlUMDY6MzU6NDEuMjIyNjU0OTM5WiJ9": happened during read: Digest did not match, expected sha256:ea5ddf9d8bc93e31d35d234af37f14773cd78efdca835c6303de2556966e12e8, got sha256:01ae2daa9fbc8a7d8c34942393c8717273c20bcbc62455861c3d4de476e21bab

Describe the results you expected

I expected push to docker.io to complete successfully.

podman info output

root@freebsd:~ # podman info
host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers: null
  cgroupManager: ""
  cgroupVersion: ""
  conmon:
    package: conmon-2.1.12_4
    path: /usr/local/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 96.96876964279447
    systemPercent: 1.7248095800129426
    userPercent: 1.2857101833757745
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: freebsd
    version: "14.2"
  eventLogger: file
  hostname: freebsd
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 14.2-RELEASE
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 265224192
  memTotal: 2104791040
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: ocijail
    package: ocijail-0.4.0_1
    path: /usr/local/bin/ocijail
    version: ocijail version 0.4.0
  os: freebsd
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /var/run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: ""
    rootless: false
    seccompEnabled: false
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2135490560
  swapTotal: 2147483648
  uptime: 0h 57m 34.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/local/etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: zfs
  graphOptions: {}
  graphRoot: /var/db/containers/storage
  graphRootAllocated: 514787434496
  graphRootUsed: 3943067648
  graphStatus:
    Compression: lz4
    Parent Dataset: zroot/ROOT/default
    Parent Quota: "no"
    Space Available: "510844366848"
    Space Used By Parent: "4349345792"
    Zpool: zroot
    Zpool Health: ONLINE
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /var/run/containers/storage
  transientStore: false
  volumePath: /var/db/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1738216454
  BuiltTime: Thu Jan 30 07:54:14 2025
  GitCommit: ""
  GoVersion: go1.21.13
  Os: freebsd
  OsArch: freebsd/amd64
  Version: 5.2.5

root@freebsd:~ # pkg info podman
podman-5.2.5_2
Name           : podman
Version        : 5.2.5_2
Installed on   : Sun Feb  9 07:44:28 2025 EET
Origin         : sysutils/podman
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : APACHE20
Maintainer     : [email protected]
WWW            : https://podman.io/
Comment        : Manage Pods, Containers and Container Images
Shared Libs required:
        libgpgme.so.11
Annotations    :
        FreeBSD_version: 1401000
        build_timestamp: 2025-01-30T05:54:04+0000
        built_by       : poudriere-git-3.4.2
        port_checkout_unclean: no
        port_git_hash  : dddb9bab3
        ports_top_checkout_unclean: no
        ports_top_git_hash: 2e0e0c62c
        repo_type      : binary
        repository     : FreeBSD
Flat size      : 102MiB
Description    :
Podman (Pod Manager) is a fully featured container engine that is a
simple daemonless tool.  Podman provides a Docker-CLI comparable
command line that eases the transition from other container engines
and allows the management of pods, containers and images.  Simply put:
alias docker=podman.

Podman uses Buildah(1) internally to create container images.  Both
tools share image (not container) storage, hence each can use or
manipulate images (but not containers) created by the other.

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

None, its just a default FreeBSD 14.2 updated to 9feb2025.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[dragospacher]

just a small title change

[Luap99]

Please try with the latest podman version