npm spining in recvfrom() when run on s390x via qemu

[hardillb]

Version

v18.18.2

Platform

Linux c8fdac6bf542 6.4.15-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 7 00:25:01 UTC 2023 s390x Linux

Subsystem

No response

What steps will reproduce the bug?

docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v ./qemu.log:/qemu.log --rm node:18-alpine npm install node-red@3.1.0

How often does it reproduce? Is there a required condition?

Every time

What is the expected behavior? Why is that the expected behavior?

npm to complete run on emulated s390x as this is the standard way to build s390x containers when no access to s390x hardware.

What do you see instead?

Hang and 100% cpu from the npm process

Additional information

See nodejs/docker-node#1973 for qemu strace output

[lpinca]

@nodejs/platform-s390

[mhdawson]

@hardillb have you tried this with the other non-Alpine containers?

[mhdawson]

@hardillb I assume there must be some additional setup required in terms of installing QEMU etc. Can you provide the platform you were running on as well as the full set of steps for setup required before you run the docker line.

Running on my x86 machine just gave me:

[midawson@drx-hemera ~]$ docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v ./qemu.log:/qemu.log --rm node:18-alpine npm install node-red:3.1.0
docker: Error response from daemon: create ./qemu.log: "./qemu.log" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.

[hardillb]

The latest version of docker will use relative paths for volume mounts, but the older versions need a full path so the full instructions are:

• touch qemu.log to create an empty file
• docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v $(pwd)/qemu.log:/qemu.log --rm node:18-alpine npm install node-red@3.1.0 here $(pwd) should expand to the current working directory.

(also changed the node-red:3.1.0 to node-red@3.1.0)

I get the same behaviour swapping out the 18-alpine tag with the 20-alpine container, but I don't see it with the Debian based builds (e.g. 20-slim) which strongly suggests this may be an alpine/muslc based issue.

I also get the same behaviour when running on a Fedora 38 or an Ubuntu based host OS (and on GitHub Actions which is where I originally saw this)

[bnoordhuis]

Do the host and guest system differ in endianness? I'm guessing the answer is 'yes' because s390x is big endian and the host is presumably x86_64? You're quite possibly hitting qemu/qemu@44cf6731d6b and that's fixed in qemu 8.

[hardillb]

All the hosts have been AMD64 based so far.

I'll try and work out how to get a copy of qemu 8 and docker to test. Assuming it is then the problem may be how to get this all deployed to GH Actions with the actions from Docker.

[hardillb]

But also this used to work, so given the fix is to do with IPv6 addresses, had npmjs.org rolled out IPv6 recently?

I know that NodeJS changed it's default behaviour about which address it presents first if both IPv4 & IPv6 addresses are returned via DNS.

I know there is the --dns-result-order=ipv4first command line argument, is there an environment variable (which will be easier to get in scope) that will do the same I can test?

[hardillb]

I've just tested with the following:

docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v $(pwd)/qemu.log:/qemu.log --rm node:18-alpine node --dns-result-order=ipv4first /usr/local/bin/npm install node-red@3.1.0

Which should force the use of an IPv4 address and I get the same hang, so that looks like it may not be the mentioned qemu issue.

[davidhay1969]

@hardillb apologies for nosing into this issue, but, to check my understanding, I believe you're running this on an x86-64 box, using QEMU to provide the s390x emulation ? Is that the case ?

Initially, I read it that you were running this on an actual s390x box but, of course, you'd not then need QEMU ?

I say that because, for me, using Ubuntu 20.04.6 LTS with kernel 5.4.0-125-generic on an IBM Z ( s390x ) Virtual Server instance (VSI), Docker doesn't hang: -

docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v $(pwd)/qemu.log:/qemu.log --rm node:18-alpine npm install node-red@3.1.0

but instead returns: -

npm ERR! Tracker "idealTree" already exists
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-10-30T14_27_59_072Z-debug-0.log

with: -

docker --version

Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1

So, just to check, this is QEMU emulating s390x where the problem arises ?

[hardillb]

@davidhay1969 yes, this is when using the docker buildx build multi-arch capabilities on x86_64 (the default GH Action runner) which uses Qemu to run the s390x build.

[davidhay1969]

Hi @hardillb OK, so I used the following setup: -

• Quad core x86-64
• Ubuntu 22.04.3 LTS
• Kernel 5.15.0-79-generic
• Docker Docker version 24.0.7, build afdd53b
• QEMU dpkg --list | grep qemu

ii  qemu                                  1:6.2+dfsg-2ubuntu6.15                  amd64        fast processor emulator, dummy package

and setup multi-platform support via Multi-platform images

docker run --privileged --rm tonistiigi/binfmt --install all

which returned: -

{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/mips64",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "python3.10",
    "qemu-aarch64",
    "qemu-arm",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}

With that in place, I was able to start a container from the node:18-alpine image: -

docker run --platform linux/s390x -it node:18-alpine sh

and confirm the s390x emulation: -

uname -a

Linux d95867f9014d 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 s390x Linux

I was then able to run the Node image as per your suggestion above : -

touch qemu.log

docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v $(pwd)/qemu.log:/qemu.log --rm node:18-alpine npm install node-red@3.1.0

which, as per my experiment on a real s390x VSI returns: -

npm ERR! Tracker "idealTree" already exists

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-10-30T15_47_18_622Z-debug-0.log

and qemu.log gets populated full 'o stuff

If I time the docker run command, I see: -

real	0m16.148s
user	0m0.026s
sys	0m0.037s

So it takes a wee while but doesn't hang

If I prune Docker via docker system prune -a --volumes it takes a while longer, including the image pull: -

time docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v $(pwd)/qemu.log:/qemu.log --rm node:18-alpine npm install node-red@3.1.0

Unable to find image 'node:18-alpine' locally
18-alpine: Pulling from library/node
47539bffe0f4: Pull complete
48ff038cd430: Pull complete
8302ffd6337d: Pull complete
4261b725713f: Pull complete
Digest: sha256:435dcad253bb5b7f347ebc69c8cc52de7c912eb7241098b920f2fc2d7843183d
Status: Downloaded newer image for node:18-alpine
npm ERR! Tracker "idealTree" already exists

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-10-30T15_51_27_865Z-debug-0.log

real	0m29.484s
user	0m0.032s
sys	0m0.051s

Apart from Multi-platform images I also reviewed Emulating a big-endian s390x with QEMU, especially given that I so rarely use docker buildx etc. having access to a bunch o' s390x infrastructure

Not sure whether this is of ANY use ?

[davidhay1969]

@hardillb were my ramblings of any small use ?