dockerd cannot run inside the vagrant docker provider due to iptables

[ckmorris]

When provisioning the container on M1 in x86_64 mode, an error is received:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

service docker start

Stdout from the command:


Stderr from the command:

Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

I jumped into the console and tried to run dockerd to see what the output was, and it was an issue with iptables being converted to nftables rather than legacy mode, and after updating iptables to use iptables-legacy:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

A different error is encountered: iptables v1.8.7 (legacy): can't initialize iptables table 'nat': iptables who? (do you need to insmod?)

As a result, it is impossible to provision docker containers inside the machine.

[rofrano]

Confirmed. I'm seeing the same error with the rofrano/vagrant-provider:ubuntu image on both Intel and ARM platforms. Apparently they no longer work even when the architectures match!

The rofrano/vagrant-provider:debian image works fine with Intel-on-Intel and ARM-on-ARM but it gives the same error when used as Intel-on-ARM.

I'm not sure what has changed but I assume it is Docker since Docker has broken this environment several times before with their updates. I'll have to see when I have time to look into this because it is working fine on Apple M1 with the ARM Debian image which is what I mostly use it for.

If you get any further let me know. If I need to make changes to the base image just let me know or fix it and create a Pull Request. I'm good with either way.

[vinalca]

Not a very useful comment, but just to add my voice that I am interested in running Ubuntu x86 on Apple M1 and I am seeing the same issue.

[rogerbramon]

I'm very interested too! Is there anything we can do to help?

[rofrano]

@rogerbramon Yes, in fact there is something you can do. I've been exploring some options and just need more time to dedicate to this but this is where I'm at. If anyones has time to investigate further to help and maybe submit a pull request with a fix that would be great. That goes for @vinalca or @ckmorris as well.

I can't get Docker to run in Docker on an Ubuntu image on the Apple M1 regardless of what I do. Debian seems to be fine. Then I remembered that Visual Studio Code Remote Containers now has support for Docker in Docker. So here's what I did:

I started Visual Studio Code with the Remote Containers extensions on my Apple M1 Mac mini and configured an Ubuntu image for Docker in Docker and let it build. Amazingly it worked. Then I substituted the "official" Ubuntu image for their Microsoft image and it continued to work as long as I created a vscode userid.

So I now have Docker in Docker running in an Ubuntu container on my Apple M1 Mac mini (without Vagrant of course). The next step is to figure out what magic their install.sh script does that I need to do in our image so that it works again with Vagrant.

If any of you has some time to work on this it would be very helpful. What they do is build your image first, then they build a second image FROM your image and run their install scripts on top of it. What we need to figure out is what do those scripts do to make Docker work in an Ubuntu container.

I have been able to manually build an Ubuntu image with Docker using these scripts:
https://github.com/microsoft/vscode-dev-containers/tree/main/containers/docker-in-docker

It's 732MB in size! I'd like to try and get it a little smaller. That's kind of big for a vagrant image IMHO. The trick is, I need to figure out what to do to the base image to allow docker to install because not everyone is going to want Docker on their Ubuntu vagrant image. Otherwise maybe I'll just make a new image with is Ubuntu with Docker pre-installed? Thoughts?

Any help you can provide would be greatly appreciated. Thanks in advance.

[rofrano]

@ckmorris, @rogerbramon, & @vinalca, I have this working. I upgraded the default rofrano/vagrant-provider:ubuntu to be based on ubuntu:jammy which is 22.04 and Docker is working again. I worked through the iptables patch that Chris suggested and this time it told me to update the kernel. When I updated the base image to 22.04 it all just started working with no other changes.

Please let me know if this works for you and I will close this issue. Thanks!

[rogerbramon]

Thanks @rofrano for your time. I was actually trying the same this morning with no luck when trying to run Intel-on-ARM (using --platform=linux/arm64 on a M1 Pro). I just tried your new image with the same result. Does it work for you?

I'm currently on Docker 4.6 and MacOS 12.1. I can try to update to MacOS 12.3 tonight to see if that the issue.

This is the Vagrantfile that I use:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.hostname = "ubuntu"

  ############################################################
  # Provider for Docker on Intel or ARM (aarch64)
  ############################################################
  config.vm.provider :docker do |docker, override|
    override.vm.box = nil
    docker.image = "rofrano/vagrant-provider:ubuntu"
    docker.remains_running = true
    docker.has_ssh = true
    docker.privileged = true
    docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
    docker.create_args = ["--platform=linux/amd64", "--cgroupns=host"]     
  end  

  # Install Docker and pull an image
  config.vm.provision :docker do |d|
    d.pull_images "alpine:latest"
  end

end

Thanks!

[rofrano]

Hi @rogerbramon , I am getting the same error for an Intel VM. I forgot that you guys want to run intel VMs on M1. I only use ARM VMs and it works fine for vagrant with Ubuntu and Debian on ARM.

With your Vagrantfile I get the same original error with iptables:

iptables/1.8.7 Failed to initialize nft: Protocol not supported

I cannot get this to work with an Intel VM and I don't know how to fix this but I would be happy to accept a pull request from someone who can.

[rogerbramon]

Thanks @rofrano for checking it out. While checking the vscode repo you mentioned above, I noticed they have a note alerting that it doesn't work in cross architectures...

Emulating X86 on M1 is very convenient when the vagrant script you want to provision is assuming you have X86_64 architecture.