Default podman machine list --all-providers to true

[rhatdan]

Podman Desktop would like this to be the default, and I see little reason why it would not be the default.

Does this PR introduce a user-facing change?

podman machine --list will now default to showing all machines for all profiles.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rhatdan]

@benoitf PTAL

[benoitf]

thanks 👍 it'll help people figuring out why Podman Desktop displays some machine while the rodman CLI doesn't ( without the flag )

[baude]

I do not agree with this

[baude]

instead make libkrun the default apple provider

[jakecorrenti]

I'm not a massive fan of this either, but I suppose I can see the appeal.

Could I get some additional context as to why this change is wanted? Wouldn't always listing a mix of multiple providers be confusing for the user?

[benoitf]

the main problem is that if you create podman machines using different providers, you only see from the CLI only one machine while Podman Desktop will display them all. So it's becoming confusing

The same also if podman CLI is using by default libkrun and user wants to create (to compare/ to test applehv machine) or the opposite, the user doesn't see anymore the machine.

the same if user create a machine using CONTAINERS_MACHINE_PROVIDER prefix

we (podman desktop)'re getting multiple reports that the cli is not listing the machines.

[jakecorrenti]

To me, it makes sense to only list the current set of machines the user can successfully interact with in the CLI's current state. By "current state" I mean the provider specified in containers.conf and not prefixing the command with CONTAINERS_MACHINE_PROVIDER.

I would find it confusing to list machines, by default, that I can't interact with unless I prefixed my command with an environment variable or stopped my current machine and edited my containers.conf.

I'm not going to die on that hill, though.

[benoitf]

there is this existing RFE as well: #20702 to be able to interact easily with the providers.

[baude]

This is a long standing problem caused by a difference in how the machine provider is determined between Podman Desktop and Podman. Podman uses containers.conf (if present, otherwise uses a default) to determine the machine provider. Podman Desktop does not interact with this config file for its machine provider. This leads to user experience problems with the Podman command line after Podman Desktop creates or runs a machine with the opposite provider.

Here are some options for consideration on this problem. They are in no particular order, though I happen to favor the first.

• Podman Desktop needs to honor the containers.conf setting. If Podman Desktop overrides the current provider, then it should update the provider in containers.conf. I'm sure there will be debate amongst our team if this is desirable, but it does keep Podman and Podman Desktop in sync.
• Eliminate or deprecate one of the MacOS providers in Podman after ensuring functional parity between the two. I think this will happen naturally but it could theoretically be forced earlier.
• Merge this PR. The downside is the subsequent confusion for users who can see machines with Podman but won't be able to start, stop, rm, etc unless they edit the containers.conf file or use the environment variable. My opinion here is this does more harm than good.
• Refactor and rewrite Podman machine to open up capabilities we did not want in Podman and are reluctant to support. Downside here is it opens failure vectors that would require a lot more testing and lot more code writing. Moreover, I'm not sure after all that work, we really solve a problem or cause more.
• Add some intelligence to Podman that when you execute machine commands and we detect things going on with a different provider, we spill out errors. This would require nailing down all of these odd use cases.
• Do nothing
• Other ideas?

[benoitf]

This is a long standing problem caused by a difference in how the machine provider is determined between Podman Desktop and Podman. Podman uses containers.conf (if present, otherwise uses a default) to determine the machine provider. Podman Desktop does not interact with this config file for its machine provider. This leads to user experience problems with the Podman command line after Podman Desktop creates or runs a machine with the opposite provider.

I would say it's a false statement. Podman Desktop read that file and honor also that file to select the default behaviour. But in the UI you need to be able to select the type of the machine you want to select. So podman supports both applehv provider and libkrun provider so it's up to the user to select the provider he wants.

Else probably nobody would be able to create machines using libkrun. Also we received complaints because if the user is asking to create a machine using the Apple GPU, then we switch the config file but then people no longer see the previous machine as well.

This is why on my end I would be in favor of the list command listing really what is on the system.

The configuration file is stating

https://github.com/containers/common/blob/main/docs/containers.conf.5.md

provider=""
Virtualization provider to be used for running a podman-machine VM. Empty value is interpreted as the default provider for the current host OS.

so it's telling me that it'll use that flag to run a machine, not when listing.

and there was #20702 asking to be able to interact with machines and specifying the provider type argument so I could easily remove any machine on my system.

so for example today you can't init a machine having the same name if used by another provider, so you're like in the middle. you're using a provider but the other provider is affecting you but if you list you don't see other machines.

As user experience, I would expect a less painful experience if

podman machine ls list all machines
podman machine init default to the default provider of my configuration file (or the environment variable)
podman machine init --provider <provider> select the provider to use (would be a bonus)
podman machine stop| start|inspect inspect the machine (whatever the provider is as the name can't collide)

and of course, you can only have a single machine running at the time.

then switching the default provider from 'applehv' to 'libkrun' would be a seamless experience

[mheon]

I think the fundamental bit here is that podman machine is not intended to support more than one podman machine VM simultaneously. This is very deliberate, and intended to make things simpler (both for us to maintain, and for users who don't have to worry about which VM owns the socket they're talking to, which VM has the container they want, etc). As such our architecture is focused on a single VM and a single provider at a time. If we were going to change that, the time would have been Podman 5.0, but that's come and gone so it's going to be with us for a long time. (Alternatively, we could just get rid of one of the providers entirely, as Brent said... It'd solve a bloody lot of problems, assuming we could get one provider with support for all required features).

Given this I'm against this PR. The current behavior makes the most sense in our current world, where podman machine is very tied to a single provider at any given time. I understand that doesn't match the UI model in Desktop, but I'm pretty sure we'd need another major rework of the podman machine codebase and a breaking change release of Podman to be able to fully match what Desktop wants there. That would be a substantial time and resource investment, and we hadn't been planning on a Podman 6 before next year.

I don't see an easy short-term solution. The best option may be for Desktop to write containers.conf with a new provider when the active VM is switched, which does not fully address the problem but at least ensures that the in-use VM can be interacted with by podman machine commands. Documentation around this in both Podman and Podman Desktop can also be improved to try and address questions.

[rhatdan]

The idea here is not to support two machines at the same time.
But my understanding of the problem is if I have two machines on my MAC and one is libkrun and the other is vfkit, then podman machine ls only lists on of them, I guess based on the settings in containers.conf.

If I want to remove one of the machines, I would like to be able to see them in the ls command.

Hiding them behind a flag makes little sense to me.

I Fully agree that running two machines at the same time would be difficult and would need a big justification.

We hit this because RamaLama is trying to identify if the podman machine is running with libkrun, and the machine was not shown.

Bottom line for me though, is I would want the default for podman machine to move to libkrun. This has better support for GPU Sharing which is a Huge importance right now. Docker is moving this way using krunkit. I also believe libkrun has better virtiofs support. Next year Apple might release a version of the OS with GPU sharing to VMs, so this decision could be rethought at that time.

[mheon]

What you are describing does not work right now, because provider switching is required before any commands on a VM not using the current provider will function. When podman machine has provider set to libkrun we can only interact with libkrun VMs; when podman machine is in applehv mode, we can only interact with applehv VMs. Running podman machine rm on an applehv machine when using the libkrun provider will error. Changing that is a major rework of the architecture of Machine and probably a Podman 6 change. Until and unless that happens changing the --all-providers default to true does not make sense in my eyes; we're showing them machines they cannot interact with. Multiple providers was never intended when we wrote machine, and still didn't make the cut when we did the Podman 5 rearchitecture.

Changing to krun as the default should be a separate discussion. I'm amenable to this for new installs, though existing installs will have to keep the current default to keep using existing machines. But I would like to here what @baude thinks about it.

[benoitf]

What you are describing does not work right now, because provider switching is required before any commands on a VM not using the current provider will function

I would like to add a caveat to this assertion.
Most of the podman cli commands are using the system connection to interact. So if I do:

CONTAINERS_MACHINE_PROVIDER=libkrun podman machine init --now

then I can launch containers, pull images, get the version of my remote podman

$ podman run quay.io/podman/hello
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob sha256:1ff9adeff4443b503b304e7aa4c37bb90762947125f4a522b370162a7492ff47
Copying config sha256:83fc7ce1224f5ed3885f6aaec0bb001c0bbb2a308e3250d7408804a720c72a32
Writing manifest to image destination
!... Hello Podman World ...!

         .--"--.
       / -     - \
      / (O)   (O) \
   ~~~| -=(,Y,)=- |
    .---. /`  \   |~~
 ~/  o  o \~~~~.----. ~~
  | =(X)= |~  / (O (O) \
   ~~~~~~~  ~| =(Y_)=-  |
  ~~~~    ~~~|   U      |~~

Project:   https://github.com/containers/podman
Website:   https://podman.io
Desktop:   https://podman-desktop.io
Documents: https://docs.podman.io
YouTube:   https://youtube.com/@Podman
X/Twitter: @Podman_io
Mastodon:  @[email protected]

$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
95f1cca64624 quay.io/podman/hello:latest /usr/local/bin/po... 23 seconds ago Exited (0) 23 seconds ago blissful_yonath

$ podman pod create --label myFirstPod
a9d87a344527970c39f79d049cec0a64ad5681c0f4627c1eaef1de48edfc61fe

so all commands are working except `machine` namespace` as when I list a machine it'll tell me there is none while there is one running just that I don't see it. So first feedback from a user is : where podman is connected ?! I don't have a machine but everything is working.


and if enter `podman machine reset` it's telling me that it'll remove a machine from libkrun provider that I have not seen.

$ podman machine reset
Warning: this command will delete all existing Podman machines
and all of the configuration and data directories for Podman machines

The following machine(s) will be deleted:

NAME PROVIDER
podman-machine-default libkrun

If `libkrun` is becoming a default provider it would also mean that if I am a user of podman and I only update the CLI, I won't see anymore my existing podman machine using `applehv` without requiring a flag `--all-providers` that probably nobody is really aware of.

[mheon]

I apologize for the misstatement; I meant machine commands. Full access to the API is available so long as it's being forwarded, so provider does not matter for these commands.

This does not, however, alter my point. With the way the machine commands (and more specifically their backend) are structured, we cannot support multi-provider operation. We have hacked this into certain commands like machine reset and machine ps --all. Hacking it into every single command is not a viable long-term solution - we need this to be supportable and maintainable. And there are definitely questions about what the right way to do this would be - do we need a unified location for configs? A second, global listing of what machines are active (that could get out of sync if removal or addition fail at awkward points)? An architectural discussion will need to happen before we implement so we have good answers to what the final solution will look like.

In brief, this sounds like something you'll want to raise as a future priority. After evaluating difficulty and priority, we can see where it will slot in, but I continue to suspect this will be a breaking change and we have no immediate plans for a Podman 6.

[baude]

@rhatdan can you clarify on the ramalama point you made earlier? do you observe this because Podman Desktop is in play here or purely with the CLI?

@rhatdan did you see the list of solutions I provided? Is it safe to say you would like us to deprecate vfkit in favor of libkrun or just swap defaults?

[baude]

For the record here, the only bullet proof solution to this is use the containers.conf so both podman and podman desktop can remain in sync.

[rhatdan]

RamaLama has switched to checking --all-providers by default. So this argument from a RamaLama point of view is mute.

From a Human point of view podman machine list not listing all of the machines, is a bug,

 $ podman machine list --help
List machines

Description:
  List managed virtual machines.

...

       │ list    │ podman-machine-list(1)    │ List virtual machines 

If this can not be fixed until 6.0 I guess I have to live with that.

As a user, I would expect if I switched from vfkit to libkrun and then could not remove the vfkit VM without switching back, I would think the tool is very broken.

[mheon]

Switching between providers was not intended to be a common or routine thing, so we didn't architect things for convenience around it. I would say it was a missed opportunity in 5.0, but we already had a lot on our plate that release so I don't think we could have crammed this in as well.