Unable to connect to Podman socket - A socket operation encountered a dead network.

[nddipiazza]

On Windows Server 2022.

Under the Service account user, I'm able to log in to the Windows Server as this user and set up podman like the following

podman machine init --user-mode-networking 
podman machine start
podman pull quay.io/podman/hello

Great I can see that it's running:

API forwarding listening on: npipe:////./pipe/docker_engine

Then from another process I'm testing this out via an Azure devops pipeline that does this:

podman pull quay.io/podman/hello

Here's the result:

Starting: PowerShell
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.230.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'c:\Agent\_work\_temp\d82f425a-2aa8-4891-824e-878c190bea53.ps1'"
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v5.3.1/libpod/_ping": dial unix /run/podman/podman.sock: connect: A socket operation encountered a dead network.
##[error]PowerShell exited with code '1'.
Finishing: PowerShell

Why is it unable to find the socket?

Is there some way to specify the IP address and Port via DOCKER_HOST? Or something?

[baude]

@l0rd could you respond to this ?

[l0rd]

Why is it unable to find the socket

I can't reproduce the problem so it's hard to tell. @nddipiazza here is my setup please see what are the differences, that would help me reproducing the problem:

• I am on Windows 11 Pro (but that we our CI runs on Windows Server 2022 and I never spot any difference)
• Podman is running on WSL v2.3.26.0 on my setup
• Podman version is the latest one (5.4 but 5.3.x should be the same)
• I ran podman init --user-mode-networking and podman start as specified in the issue description
• I tested the command powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". ' C:\Users\mario\test.ps1'" and the file C:\Users\mario\test.ps1 content is:

podman pull quay.io/podman/hello

and the scripts runs successfully.

Is there some way to specify the IP address and Port via DOCKER_HOST? Or something?

There should be no need to specify it but you can add $Env:DOCKER_HOST = 'npipe:////./pipe/podman-machine-default' (replace podman-machine-default with the name of the podman machine that's running) at the beginning of the powershell script.

[nddipiazza]

@l0rd I am running from Azure Devops on-prem trying to utilize Podman within the pipeline.

I am connecting with remote desktop into the server with user nick
then running the podman machine commands to initialize the machine as that user.

This user runs the Azure devops agent is azuredevopsaccount - which is not a real windows user it's a service account

I'm guessing the problem is permissions as this does not own permission. It fails to find the podman machine in this situation.

Am I just using this in an unapproved/unintended way and suffering the consequences?

[l0rd]

@nddipiazza, thank you for the details; I was able to reproduce the problem if I open a powershell terminal as a different user on my local windows.

Setting $Env:DOCKER_HOST = 'npipe:////./pipe/podman-machine-default' doesn't have any effect.
Setting $Env:CONTAINER_HOST = 'npipe:////./pipe/podman-machine-default' I get a different error "Error: unable to create connection. "npipe" is not a supported schema".

I will continue investigating.

[l0rd]

@nddipiazza to address your issue you have 2 choices:

• delete the machine that you created as nick, open a new terminal as azuredevopsaccount and run machine init and machine start from it
• keep the machine you currently have and, from a new terminal as azuredevopsaccount generate an ssh keypair, then add the public key into nick machine and use podman system connection add as documented here to create an SSH connection
  for azuredevopsaccount.

[nddipiazza]

delete the machine that you created as nick, open a new terminal as azuredevopsaccount and run machine init and machine start from it

trying this but when I do it this way the network is unavailable on the podman machine.

so my attempt is to take ADO pipeline and do this:

• podman machine init
• podman machine start
• attempt to do a podman pull

So something about running this as this "headless" user makes it so I cannot access the internet.

I am able to do the podman machine init, machine start, podman pull when I am logged in via a remote desktop session to Windows, but not when running from the pipeline which would be from a background process not backed by a windows desktop session.

[nddipiazza]

delete the machine that you created as nick, open a new terminal as azuredevopsaccount and run machine init and machine start from it

I like this option so much I want to eliminate it as an absolute impossibility before abandoning it. Because this does not require me actually having a "keep-alive" windows session logged in as a user. It can run headless.

[nddipiazza]

[l0rd]

I don't know why that's happening. In particular, it's surprising that the ADO service can successfully run podman machine init (which requires access to the external network to download the machine image), but that podman pull fails.

Anyway, I will try to run a headless service and reproduce the problem, although it may not be easy on Windows 11.

For your scenario, my recommendation would be to run Podman in a VM that is always up (and you may prefer Hyper-V rather than WSL for that reason) and connect to it from the ADO pipeline. Creating and deleting Podman machines as part of ADO pipelines makes your CI slower and more prone to errors.

[nddipiazza]

I would love to use that approach with Linux dedicated vm stuff - because it would make this so much easier. But I'm stuck on Windows VMs because they are already required for other testing purposes and I want to better utilize them by adding podman

i will not start podman macine each time via pipeline, that will be started outside of the pipeline as a windows service along side the Azure devops build agent. the key is that it will be running as the same service account

i am going to fire up a bare metal windows sever 2022 instance and see if i can simulate this same scenario

[l0rd]

You may want to try to create a Podman VM using Hyper-V rather than WSL, so it lasts even after the user who did the machine init logs out.

In the containers.conf:

[machine]
provider="hyperv"

[nddipiazza]

OK i'll give that a shot today.

[nddipiazza]

i created the system env variable

echo $env:CONTAINERS_MACHINE_PROVIDER
hyperv

[nddipiazza]

ok i re-installed wsl installed it then chose hyper-v. will let you know if it works

[nddipiazza]

oops didn't mean to close this

[nddipiazza]

@l0rd can you re-open this? i have fat fingers and apparently the comment button wasn't green enough