Apache Proxy to Docker Container Causes 408 After a Short Delay

[abegosum]

Current Behavior

I'm attempting to reverse-proxy via Apache on the host machine to a netbox docker container. The initial request seems to work, but after some delay (I typically wait 5 mins for testing), the Apache server will ONLY respond with 408 timeouts. This is unusual, considering I frequently use Apache as a reverse proxy without issue.

Below is a sanitized version of my Apache vhost configuration.

<VirtualHost *:443>
    ServerName netboxlab.mydomain.org
    SSLEngine on
    SSLCertificateFile /path/to/cert.pem
    SSLCertificateKeyFile /path/to/key.pem
    SSLCertificateChainFile /path/to/chain.pem

    ProxyPreserveHost Off
    SSLProxyEngine Off
    ProxyRequests Off
    ProxyPass '/' 'http://127.0.0.1:8000/'
    ProxyPassReverse '/' 'http://127.0.0.1:8000/'

    CustomLog "/logs/apache2/netbox/logs/access_log" combined
    ErrorLog "/logs/apache2/netbox/logs/error_log"
</VirtualHost>

The container logs seem to indicate that Apache is no longer proxying requests.

Things I've tried that don't fix the issue:

• Setting proxytimeout to 120
• Setting ping on proxypass to 5
• Setting keepalive on proxypass to on
• Pointing proxypass and proxyreversepass to localhost
• Pointing proxypass and proxyreversepass to the hostname of the machine (with that added to the ALLOWED_HOSTS variable in the docker container)
• Changed the MPM module from prefork to worker mode

My environment:

Apache Version: 2.4.6
CentOS Version: 7.9.2009
Netbox-Docker Project Tag: 1.5.1
Netbox Image Tag (in docker-compose.override.yml): v3.1

Env vars overridden in .env.prod: DB_PASSWORD, SECRET_KEY, SKIP_SUPERUSER (set to 'true'), POSTGRES_PASSWORD

Expected Behavior

I expect the proxy to continue working after the initial load.

Docker Compose Version

docker-compose version 1.29.2, build 5becea4c

Docker Version

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:44:05 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The git Revision

58a1579

The git Status

# HEAD detached at 1.5.1
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.env.prod
nothing added to commit but untracked files present (use "git add" to track)

Startup Command

docker-compose up -d

NetBox Logs

Attaching to netbox-docker_netbox_1
netbox_1               | ↩️ Skip creating the superuser
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/000_users.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/010_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/015_object_permissions.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_custom_fields.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_tags.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/020_tenant_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/030_regions.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/030_tenants.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/040_sites.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/050_manufacturers.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/060_device_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/070_rack_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/075_locations.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/080_racks.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/090_device_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/100_platforms.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/130_cluster_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/135_cluster_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/135_clusters.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/140_clusters.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/140_devices.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/150_rirs.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/160_aggregates.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/165_cluster_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/175_route_targets.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/180_vrfs.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/190_prefix_vlan_roles.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/200_vlan_groups.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/210_vlans.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/220_prefixes.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/230_virtual_machines.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/240_virtualization_interfaces.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/250_dcim_interfaces.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/260_ip_addresses.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/270_primary_ips.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/280_custom_links.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/280_providers.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/290_circuit_types.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/290_webhooks.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/300_circuits.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/320_services.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/330_power_panels.py
netbox_1               | ▶️  Running the startup script /opt/netbox/startup_scripts/340_power_feeds.py
netbox_1               | ✅ Initialisation is done.
netbox_1               | ⏳ Waiting for control socket to be created... (1/10)
netbox_1               | 2021/12/28 18:21:41 [warn] 9#9 Unit is running unprivileged, then it cannot use arbitrary user and group.
netbox_1               | 2021/12/28 18:21:41 [info] 9#9 unit started
netbox_1               | 2021/12/28 18:21:41 [info] 17#17 discovery started
netbox_1               | 2021/12/28 18:21:41 [notice] 17#17 module: python 3.9.5 "/usr/lib/unit/modules/python3.unit.so"
netbox_1               | 2021/12/28 18:21:41 [info] 9#9 controller started
netbox_1               | 2021/12/28 18:21:41 [notice] 9#9 process 17 exited with code 0
netbox_1               | 2021/12/28 18:21:41 [info] 19#19 router started
netbox_1               | 2021/12/28 18:21:41 [info] 19#19 OpenSSL 1.1.1l  24 Aug 2021, 101010cf
netbox_1               | ⚙️ Applying configuration from /etc/unit/nginx-unit.json
netbox_1               | 2021/12/28 18:21:42 [info] 23#23 "netbox" application started
netbox_1               | ✅ Unit configuration loaded successfully
netbox_1               | 2021/12/28 18:21:44 [notice] 9#9 process 15 exited with code 0
netbox_1               | 2021/12/28 18:21:48 [info] 32#32 "netbox" application started
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET / HTTP/1.1" 200 86116 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-external.css HTTP/1.1" 200 286568 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-light.css HTTP/1.1" 200 493648 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox-dark.css HTTP/1.1" 200 788892 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:50 +0000] "GET /static/netbox.js HTTP/1.1" 200 374756 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox_logo.svg HTTP/1.1" 200 4719 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox_icon.svg HTTP/1.1" 200 835 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/netbox-print.css HTTP/1.1" 200 1623714 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:51 +0000] "GET /static/materialdesignicons-webfont-KSYPMDN6.woff2?v=5.9.55 HTTP/1.1" 200 325244 "http://localhost:8000/static/netbox-external.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:21:54 +0000] "GET /login/ HTTP/1.1" 200 8228 "http://localhost:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:22:08 +0000] "POST /login/ HTTP/1.1" 302 0 "http://localhost:8000/login/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 172.25.0.1 - - [28/Dec/2021:18:22:09 +0000] "GET / HTTP/1.1" 200 142800 "http://localhost:8000/login/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | 2021/12/28 18:24:09 [notice] 9#9 process 32 exited with code 0
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 18 exited with code 0
netbox_1               | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox_1               | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 19 exited with code 0
netbox_1               | 2021/12/28 20:01:49 [notice] 9#9 process 23 exited with code 0

Content of docker-compose.override.yml

version: '3.4'
  
x-servicebase: &servicebase
  restart: always

x-netboxbase: &netboxbase
  <<: *servicebase
  image: netboxcommunity/netbox:v3.1
  env_file:
    - env/netbox.env
    - .env.prod

services:
  netbox:
    <<: *netboxbase
    ports:
      - 8000:8080
  netbox-worker:
    <<: *netboxbase
  netbox-housekeeping:
    <<: *netboxbase

  postgres:
    <<: *servicebase
    env_file:
      - env/postgres.env
      - .env.prod

  redis:
    <<: *servicebase
  redis-cache:
    <<: *servicebase

[abegosum]

While not a solution, I have identified a workaround.

The 408 errors seem to be related to some interaction between the Python app and the Apache proxy connections. Apache typically reuses these connections for efficiency's sake, but something about that reuse is causing the error. At least in our environment, disabling reuse of proxy connections has caused the issue to go away:

ProxyPass '/' 'http://127.0.0.1:8000/' disablereuse=on

[nawlbergs]

this is happening for me as well... after a few minutes... refreshing my browser just spins... Restarting container does nothing and I have to restart docker desktop to get it working again. (oddly after restart... it shows my postgres db still running)... i kill that then restart container and it works again.. but it was consistently going into a non-responsive state after a few minutes.

[marcquark]

Could be related to #854 - perhaps the workaround posted there is helpful?

[supporttv]

For me the reviewed config worked, until a "CSRF verification failed. Request aborted." occurred after login. Setting the below X-Forwarded lines solved all proxy issues:

<VirtualHost *:443>
ServerName netboxlab.mydomain.org
SSLEngine on
SSLCertificateFile /path/to/cert.pem
SSLCertificateKeyFile /path/to/key.pem
SSLCertificateChainFile /path/to/chain.pem

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

ProxyPreserveHost Off
SSLProxyEngine Off
ProxyRequests Off
ProxyPass '/' 'http://127.0.0.1:8000/' disablereuse=on
ProxyPassReverse '/' 'http://127.0.0.1:8000/'

CustomLog "/logs/apache2/netbox/logs/access_log" combined
ErrorLog "/logs/apache2/netbox/logs/error_log"