Bootc disk build fails on a rootful Podman machine when the host's root user container storage is mounted.

[skycastlelily]

Issue Description

my testcode add "-v", /var/lib/containers/storage:/var/lib/containers/storage when perform "podman", "machine", "init", "--rootful", the podman machine start successfully, but building bootc disk from a container image using bootc image builder in that podman machine throws out the error:

Command 'podman run --rm --privileged -v /var/lib/containers/storage:/var/lib/containers/storage --security-opt label=type:unconfined_t -v /var/tmp/tmt/run-068/plans/thread/provision/default-0:/output [quay.io/centos-bootc/bootc-image-builder:latest](http://quay.io/centos-bootc/bootc-image-builder:latest) build --type qcow2 --rootfs xfs --local [quay.io/fedora/fedora-bootc:41](http://quay.io/fedora/fedora-bootc:41)' returned 125.
  # stderr (2/2 lines)
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
  Error: unable to connect to Podman socket: Get "http://d/v5.4.0/libpod/_ping": ssh: rejected: connect failed (open failed): ssh://[root@127.0.0.1:33047/run/podman/podman.sock](http://root@127.0.0.1:33047/run/podman/podman.sock)

I could ssh into that machine after that, /var/lib/containers/storage is mounted successfully,
but I got an error when I run "sudo podman image ls"

(dev) lnie@hpe-bl460cgen8-01:~/tmt$ podman machine ssh podman-machine-tmt
Connecting to vm podman-machine-tmt. To close connection, use `~.` or `exit`
Fedora CoreOS 41.20250215.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[systemd]
Failed Units: 2
 podman.service
 podman.socket

root@localhost:~# ls /var/lib/containers/storage/
db.sql  defaultNetworkBackend  libpod  overlay  overlay-containers  overlay-images  overlay-layers  secrets  storage.lock  tmp  userns.lock  volumes
core@localhost:~$ sudo podman image ls
Error: faccessat /var/lib/containers/storage/libpod/bolt_state.db: permission denied

Steps to reproduce the issue

Steps to reproduce the issue

1. _init_podman_machine
   2._build_bootc_disk

here the relative part of code:

    def _build_bootc_disk(self, containerimage: str, image_builder: str, rootfs: str) -> None:
        """
        Build the bootc disk from a container image using bootc image builder
        """

        self._logger.debug("Build bootc disk image.")

        tmt.utils.Command(
            "podman",
            "run",
            "--rm",
            "--privileged",
            "-v",
            f'{CONTAINER_STORAGE_DIR}:{CONTAINER_STORAGE_DIR}',
            "--security-opt",
            "label=type:unconfined_t",
            "-v",
            f"{self.phase_workdir}:/output",
            image_builder,
            "build",
            "--type",
            "qcow2",
            "--rootfs",
            rootfs,
            "--local",
            containerimage,
        ).run(
            cwd=self.phase_workdir,
            stream_output=True,
            logger=self._logger,
            env=PODMAN_ENV if self._rootless else None,
        )

    def _init_podman_machine(self) -> None:
        try:
            tmt.utils.Command("podman", "machine", "rm", "-f", PODMAN_MACHINE_NAME).run(
                cwd=self.phase_workdir, stream_output=True, logger=self._logger
            )
        except BaseException:
            self._logger.debug("Unable to remove existing podman machine (it might not exist).")

        self._logger.debug("Initialize podman machine.")
        # fmt: off
        tmt.utils.Command(
            "podman", "machine", "init", "--rootful",
            "--disk-size", f"{DEFAULT_PODMAN_MACHINE_DISK_SIZE.magnitude}",
            "--memory", f"{DEFAULT_PODMAN_MACHINE_MEM.magnitude}",
            "--cpus", f"{DEFAULT_PODMAN_MACHINE_CPU}",
            "-v", f"{DEFAULT_TMP_PATH}:{DEFAULT_TMP_PATH}",
            #"-v", "$HOME:$HOME",
            #"-v", f'{CONTAINER_STORAGE_DIR}:{CONTAINER_STORAGE_DIR}',
            "-v", f'/usr/sbin:{CONTAINER_STORAGE_DIR}',
            PODMAN_MACHINE_NAME,
        ).run(cwd=self.phase_workdir, stream_output=True, logger=self._logger)
        # fmt: on

        self._logger.debug("Start podman machine.")
        tmt.utils.Command("podman", "machine", "start", PODMAN_MACHINE_NAME).run(
            cwd=self.phase_workdir, stream_output=True, logger=self._logger
        )

Describe the results you received

Command 'podman run --rm --privileged -v /var/lib/containers/storage:/var/lib/containers/storage --security-opt label=type:unconfined_t -v /var/tmp/tmt/run-068/plans/thread/provision/default-0:/output quay.io/centos-bootc/bootc-image-builder:latest build --type qcow2 --rootfs xfs --local quay.io/fedora/fedora-bootc:41' returned 125.
# stderr (2/2 lines)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cannot connect to Podman. Please verify your connection to the Linux system using podman system connection list, or try podman machine init and podman machine start to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v5.4.0/libpod/_ping": ssh: rejected: connect failed (open failed): ssh://root@127.0.0.1:33047/run/podman/podman.sock

Describe the results you expected

bootc-image-builder successfully build the disk in the podman machine

podman info output

(dev) lnie@fedora:~/tmt$ podman info
host:
  arch: amd64
  buildahVersion: 1.41.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 97.55
    systemPercent: 0.61
    userPercent: 1.84
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2040
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.16.5-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 5687119872
  memTotal: 33332322304
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 6832902144
  swapTotal: 8589930496
  uptime: 394h 49m 49.00s (Approximately 16.42 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/lnie/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/lnie/.local/share/containers/storage
  graphRootAllocated: 510405902336
  graphRootUsed: 413151576064
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 69
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/lnie/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.1
  BuildOrigin: Fedora Project
  Built: 1756944000
  BuiltTime: Thu Sep  4 08:00:00 2025
  GitCommit: 1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.1


(dev) lnie@fedora:~/tmt$ rpm -q podman

podman-5.6.1-1.fc42.x86_64

(dev) lnie@fedora:~/tmt$ podman version
Client:        Podman Engine
Version:       5.6.1
API Version:   5.6.1
Go Version:    go1.24.6
Git Commit:    1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
Built:         Thu Sep  4 08:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64
(dev) lnie@fedora:~/tmt$ podman info
host:
  arch: amd64
  buildahVersion: 1.41.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 97.55
    systemPercent: 0.61
    userPercent: 1.84
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2040
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.16.5-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6869639168
  memTotal: 33332322304
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 6829608960
  swapTotal: 8589930496
  uptime: 393h 44m 27.00s (Approximately 16.38 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/lnie/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/lnie/.local/share/containers/storage
  graphRootAllocated: 510405902336
  graphRootUsed: 413016297472
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 69
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/lnie/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.1
  BuildOrigin: Fedora Project
  Built: 1756944000
  BuiltTime: Thu Sep  4 08:00:00 2025
  GitCommit: 1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

to me this smells like you did a volume mount without selinux options and now you cannot access it; did you look in the audit log?

[Luap99]

"-v", /var/lib/containers/storage:/var/lib/containers/storage

Is this done for the podman machine init command? Because doing this for podman machine init is not something we can support I believe, the virtiofs filesystem has a few limitations and cannot deal with multiple uids/gids that are required for the container storage

[skycastlelily]

Is this done for the podman machine init command?

yeah.

Because doing this for podman machine init is not something we can support I believe, the virtiofs filesystem has a few >limitations and cannot deal with multiple uids/gids that are required for the container storage

hmmm, if podman machine can't support it, shouldn't it at least warn user when we they do that for podman machine init, or when start machine?

The error throws out in boot disk building is confusing, I locate the problem because I only add the -v line

[skycastlelily]

I just checked, actually, there is one warning throw out,but confusing.

11:05:23         out: Mounting volume... /var/tmp/tmt:/var/tmp/tmt
11:05:24         out: Mounting volume... /home/lnie:/home/lnie
11:05:24         out: Mounting volume... /var/lib/containers/storage:/var/lib/containers/storage
11:05:25         err: time="2025-09-30T11:05:25+08:00" level=warning msg="API socket failed ping test"

[Luap99]

You can open a a PR add the path here https://github.com/containers/podman/blob/main/pkg/machine/shim/host.go#L822 so the machine init command outright errors.

[skycastlelily]

Thanks for your pretty valued info , and sorry for the a bit late response, I'm on pto,and here is the mr:

#27218