Container does not respect docker's DNS resolution

[F13]

Describe the bug

The DNS resolution inside the watchtower container does not respect manually provided IPs through docker's --add-host (or docker compose's extra_hosts), nor does it correctly resolve DNS aliases provided through compose's aliases.

I am trying to provide a custom IP so that the watchtower container can connect to my private registry that is hosted on another container on the same host. Watchtower refuses to accept the custom IP that provides the docker internal network address of the host; instead, it resolves upstream to the host's actual IP address. This is not reachable due to docker networking shenanigans.

Relevant docker compose snippet to illustrate my point:

services:
  watchtower:
    image: containrrr/watchtower
    environment:
      TZ: America/Chicago
      WATCHTOWER_CLEANUP: true
      WATCHTOWER_SCOPE: foo
      WATCHTOWER_RUN_ONCE: true
    volumes:
      - ${XDG_RUNTIME_DIR}/docker.sock:/var/run/docker.sock
      - ${HOME}/.docker/config.json:/config.json
    labels:
      com.centurylinklabs.watchtower.scope: foo
    networks:
      - caddy
    extra_hosts:
      - "registry.internal=172.20.0.2"
  foo:
    image: registry.internal/homelab/foo

When running the container, however, the IP is resolved up to the host's IP, which is then not routable:

docker compose up watchtower 
[+] Running 1/0
 ✔ Container foo-watchtower-1  Recreated                                                                                                                                                                                             0.1s 
Attaching to watchtower-1
watchtower-1  | time="2025-03-07T09:06:54-06:00" level=info msg="Watchtower 1.7.1"
watchtower-1  | time="2025-03-07T09:06:54-06:00" level=info msg="Using no notifications"
watchtower-1  | time="2025-03-07T09:06:54-06:00" level=info msg="Only checking containers in scope \"foo\""
watchtower-1  | time="2025-03-07T09:06:54-06:00" level=info msg="Running a one time update."
watchtower-1  | time="2025-03-07T09:06:57-06:00" level=info msg="Unable to update container \"/foo-foo-1\": Error response from daemon: Get \"https://registry.internal/v2/\": dial tcp 192.168.1.9:443: connect: no route to host. Proceeding to next."
watchtower-1  | time="2025-03-07T09:06:57-06:00" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower-1  | time="2025-03-07T09:06:57-06:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
watchtower-1 exited with code 0

The registry.internal DNS name resolves to the host IP (192.168.1.9) in the outer network (i.e. the host's network), but needs to be accessed over the docker IP from watchtower's perspective.

Specifying an alias in the registry.internal container's network configuration instead results in the same scenario.

I can't test with a DNS name that doesn't resolve from the host, because the host also has to be able to resolve the registry in order to start the foo container in the first place.

Steps to reproduce

1. Host a private registry on a docker container that DNS resolves to the host's IP address
2. Attempt to use watchtower to pull from that private registry, overriding the DNS entry to point to the docker IP address

Expected behavior

The DNS resolution should respect the overrides supplied in the Docker configurations.

Screenshots

No response

Environment

• Platform: Arch Linux
• Architecture: x64
• Docker Version: Docker version 27.3.1, build ce1223035a

Your logs

watchtower-1  | time="2025-03-07T09:19:07-06:00" level=debug msg="Using scope \"foo\""
watchtower-1  | time="2025-03-07T09:19:07-06:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Making sure everything is sane before starting"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=info msg="Watchtower 1.7.1"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=info msg="Using no notifications"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=info msg="Only checking containers in scope \"foo\""
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=info msg="Running a one time update."
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Checking containers for updated images"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Retrieving running containers"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Trying to load authentication credentials." container=/foo-watchtower-1 image="containrrr/watchtower:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Got image name: containrrr/watchtower:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Checking if pull is needed" container=/foo-watchtower-1 image="containrrr/watchtower:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Setting scope for auth token" image=docker.io/containrrr/watchtower scope="repository:containrrr/watchtower:pull"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="No credentials found."
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Parsing image ref" host=index.docker.io image=containrrr/watchtower normalized=docker.io/containrrr/watchtower tag=latest
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/containrrr/watchtower/manifests/latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Found a remote digest to compare with" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg=Comparing local="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Found a match"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="No pull needed. Skipping image."
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="No new images found for /foo-watchtower-1"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Trying to load authentication credentials." container=/foo-foo-1 image="registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Loaded auth credentials for user f13, on registry registry.internal, from file /config.json"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Got image name: registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Credentials loaded"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Checking if pull is needed" container=/foo-foo-1 image="registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Built challenge URL" URL="https://registry.internal/v2/"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Could not do a head request for \"registry.internal/homelab/foo:latest\", falling back to regular pull." container=/foo-foo-1 image="registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Reason: Get \"https://registry.internal/v2/\": x509: certificate signed by unknown authority" container=/foo-foo-1 image="registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:08-06:00" level=debug msg="Pulling image" container=/foo-foo-1 image="registry.internal/homelab/foo:latest"
watchtower-1  | time="2025-03-07T09:19:11-06:00" level=debug msg="Error pulling image registry.internal/homelab/foo:latest, Error response from daemon: Get \"https://registry.internal/v2/\": dial tcp 192.168.1.9:443: connect: no route to host"
watchtower-1  | time="2025-03-07T09:19:11-06:00" level=info msg="Unable to update container \"/foo-foo-1\": Error response from daemon: Get \"https://registry.internal/v2/\": dial tcp 192.168.1.9:443: connect: no route to host. Proceeding to next."
watchtower-1  | time="2025-03-07T09:19:11-06:00" level=debug msg="This is the watchtower container /foo-watchtower-1"
watchtower-1  | time="2025-03-07T09:19:11-06:00" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower-1  | time="2025-03-07T09:19:11-06:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
watchtower-1 exited with code 0

Additional context

Based on that debug output, it seems possible that this is also related to #1282, as I use a local CA to sign my TLS certificates.

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏