[awf] network-isolation: cli-proxy cannot reach the external DIFC proxy (ENETUNREACH to host gateway) on GitHub-hosted runners, no Tailscale - variant of #6704

[raman-anyware]

Problem

From v0.27.43+ (observed on v0.27.44), in network-isolation/topology mode with the CLI proxy connecting to an external DIFC proxy on the runner host, the awf-cli-proxy container never becomes healthy: its tcp-tunnel gets ENETUNREACH dialing the host gateway, the DIFC liveness probe exhausts its retries, and the workflow fails fast — the agent is never invoked.

The same workflow on the same runner pool works on v0.27.26 (pre-rework).

Key data point for #6704: this is a GitHub-hosted ubuntu-latest runner with no Tailscale — so the route-capture mechanism validated there cannot be the cause of this variant. The isolation-mode container→host path appears broken on a stock runner.

Evidence (v0.27.44, GitHub-hosted ubuntu-latest)

[INFO] CLI proxy enabled: connecting to external DIFC proxy at host.docker.internal:18443
[WARN] Host access enabled with host.docker.internal in allowed domains
[WARN] Network-isolation: removing 1 non-portable DNS server(s) that may become unreachable
       from Docker bridge containers when host routing is modified by tools like Tailscale: 168.63.129.16
[WARN] Network-isolation: no portable DNS servers remain after filtering; falling back to 8.8.8.8, 8.8.4.4.
 Container awf-cli-proxy  Error
dependency failed to start: container awf-cli-proxy is unhealthy
[cli-proxy] DIFC proxy probe failed (attempt 1/10, diagnosis=unknown), retrying in 1s...
...
[cli-proxy] DIFC proxy probe failed (attempt 6/10, diagnosis=unknown), retrying in 30s...
[tcp-tunnel] Upstream error (::1:39718): connect ENETUNREACH 172.17.0.1:18443 - Local (0.0.0.0:0)
[ERROR] Fatal error: Error: AWF firewall failed to start: awf-cli-proxy could not connect to the
        external DIFC proxy (or exited before establishing a connection). Failing fast to avoid
        repeated in-agent retries. The agent was never invoked.

Note the third line: "Host access enabled with host.docker.internal in allowed domains" is logged, yet the tunnel still cannot reach the host — the upstream dial goes to 172.17.0.1 (the default bridge gateway), which is not reachable from the isolated topology-mode network. This suggests the #6657 host-access-with-isolation plumbing does not cover the cli-proxy → external-DIFC path: host.docker.internal resolves to the default bridge's gateway rather than the gateway of the network the cli-proxy actually sits on.

Version matrix (same workflow, same runner pool)

firewall	outcome
v0.27.26	✅ cli-proxy healthy, agent runs, reviews post
v0.27.44	❌ awf-cli-proxy is unhealthy, ENETUNREACH 172.17.0.1:18443, agent never starts

Environment

• Compiler: gh-aw v0.79.8 (sandbox.agent.version override selects the firewall version), Copilot engine 1.0.60
• Runner: GitHub-hosted ubuntu-latest (Azure DNS 168.63.129.16 auto-detected); no Tailscale, no custom host routing
• CLI proxy in external-DIFC mode (host.docker.internal:18443)
• Trigger: slash_command workflow (issue_comment), permissions: contents: read, pull-requests: read
• Runs are in a private repository; happy to share full logs/awf-resolved-config.json with maintainers.

Ask

Either resolve host.docker.internal to the gateway of the isolation-mode network the cli-proxy is attached to (or add an explicit host route/alias for it), or extend the #6657 host-access plumbing to the cli-proxy sidecar. If there is a supported flag combination that restores container→host access for the external DIFC proxy under topology mode, a Runner Doctor entry for this failure signature would also help — the current message gives no hint that the firewall version is the variable.

[raman-anyware]

Reopening: this is still broken on v0.28.0, and the PR that closed it contains no production change.

The PR that closed this shipped only a test

#7066 — "Fix cli-proxy ENETUNREACH in network-isolation mode by dual-homing it on the external bridge" — changed exactly one file: src/compose-generator.test.ts, +17/−0. No production file was touched, and the test it adds asserts the opposite of the title:

it('keeps cli-proxy on awf-net only when it targets an attached DIFC proxy', () => {
  ...
  expect(cliProxyNetworks['awf-ext']).toBeUndefined();
});

That assertion is reasonable for the case it covers — an attached DIFC proxy (difcProxyHost: 'awmg-cli-proxy:18443'), a sibling container that needs no route off the internal network. But this issue is about the external DIFC proxy on the runner host, which is the other branch, and that branch got no fix.

The production code confirms it

In v0.28.0, src/services/cli-proxy-service.ts attaches the sidecar to awf-net only — the string awf-ext does not appear in the file at all — while setting:

networks: { 'awf-net': { ipv4_address: cliProxyIp } },
extra_hosts: { 'host.docker.internal': 'host-gateway' },

host-gateway resolves to the default bridge gateway (172.17.0.1), and awf-net is internal: true, so the container has no route to it. Hence the reported connect ENETUNREACH 172.17.0.1:18443.

Reproduced on v0.28.0

GitHub-hosted ubuntu-latest, no Tailscale, gh-aw v0.79.8 with sandbox.agent.version: v0.28.0, CLI proxy in external-DIFC mode:

[INFO] CLI proxy enabled: connecting to external DIFC proxy at host.docker.internal:18443
[INFO] Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
 Network awf-net  Created
 Network awf-1786580104070_awf-ext  Created
 Container awf-cli-proxy  Error
dependency failed to start: container awf-cli-proxy is unhealthy
[tcp-tunnel] Upstream error (::1:37186): connect ENETUNREACH 172.17.0.1:18443 - Local (0.0.0.0:0)
[ERROR] Fatal error: AWF firewall failed to start: awf-cli-proxy could not connect to the external
        DIFC proxy ... The agent was never invoked.

Both networks are created; only the cli-proxy is left without a route to the host. For contrast, v0.27.26 (pre-isolation-rework) runs the identical workflow successfully on the same runner pool.

The DNS fix from #7188 is working here, incidentally — v0.28.0 logs retaining reachable non-portable DNS server(s): 168.63.129.16 rather than discarding them. It is only the cli-proxy network path that remains broken.

Who this affects

Any workflow using gh-aw's tools.github.mode: gh-proxy (equivalently the legacy features: {cli-proxy: true}) on a runner where the DIFC proxy is external. That is not the default mode, which is likely why the blast radius looks small — the default local mode starts no cli-proxy at all.

Workaround, for anyone else who lands here

Switching the workflow to tools.github.mode: local removes the sidecar entirely and the failure with it. Note that it also removes the pre-authenticated gh that gh-proxy provides, so any prompt driving gh in bash has to move onto the GitHub MCP toolsets — in local mode the agent holds no GitHub credentials.

Suggested fix

Attach the cli-proxy to EXTERNAL_BRIDGE_NAME when networkIsolation is enabled and the DIFC proxy is external, mirroring what Squid and api-proxy already do — i.e. what #7066's title describes. The existing test stays valid if it is scoped to the attached-DIFC case it names.

[raman-anyware]

Refiled as #7335, since this stayed closed and I cannot reopen it. Still reproducible on v0.28.0; the new issue carries the source-level root cause and the reproduction.