Proxy header error after version migration 4.1.0 to 4.2.0 [Questions]

[skry-dev]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.0

Erlang version used

27.3.x

Operating system (distribution) used

RedHat 8.10

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Status of node rabbit@rabbit1 ...
Runtime

OS PID: 21
OS: Linux
Uptime (seconds): 1791
Is under maintenance?: false
RabbitMQ version: 4.2.0
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: eft-rabbit@eftvrabmqt1
Erlang configuration: Erlang/OTP 27 [erts-15.2.7.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.5.4 30 Sep 2025
Erlang processes: 1915 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_auth_backend_ldap
 * rabbitmq_top
 * rabbitmq_delayed_message_exchange
 * accept
 * prometheus
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * oauth2_client
 * jose

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/quorum/rabbit@rabbit1

Config files

 * /etc/rabbitmq/advanced.config
 * /etc/rabbitmq/rabbitmq.conf
 * /etc/rabbitmq/conf.d/10-defaults.conf

Log file(s)

 * /var/log/rabbitmq/rabbit.log
 * <stdout>

Alarms

(none)

Tags

(none)

Memory

Total memory used: 0.2657 gb
Calculation strategy: rss
Memory high watermark setting: 0.7 of available memory, computed to: 11.5669 gb

reserved_unallocated: 0.0677 gb (25.47 %)
allocated_unused: 0.0581 gb (21.86 %)
other_system: 0.0288 gb (10.84 %)
code: 0.0286 gb (10.78 %)
other_proc: 0.0225 gb (8.46 %)
plugins: 0.0181 gb (6.82 %)
connection_other: 0.0069 gb (2.6 %)
binary: 0.0067 gb (2.51 %)
connection_channels: 0.0062 gb (2.32 %)
other_ets: 0.0055 gb (2.07 %)
mgmt_db: 0.0036 gb (1.36 %)
metadata_store: 0.0029 gb (1.1 %)
metrics: 0.0023 gb (0.85 %)
connection_readers: 0.002 gb (0.75 %)
connection_writers: 0.0018 gb (0.69 %)
quorum_queue_procs: 0.0016 gb (0.6 %)
atom: 0.0013 gb (0.49 %)
msg_index: 0.0006 gb (0.22 %)
metadata_store_ets: 0.0003 gb (0.11 %)
quorum_ets: 0.0002 gb (0.06 %)
mnesia: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 0, limit: 1048479

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 21.0059 gb

Totals

Connection count: 59
Queue count: 8
Virtual host count: 2

Listeners

Interface: [::], port: 15671, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 15691, protocol: https/prometheus, purpose: Prometheus exporter API over TLS (HTTPS)
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-11-07 10:57:07.001158+03:00 [notice] <0.209.0> Logging: configured log handlers are now ACTIVE
2025-11-07 10:57:07.024495+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:07.024875+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:07.039947+03:00 [info] <0.217.0> ra_coordination_log_ets: in system coordination initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,false}]
2025-11-07 10:57:07.162532+03:00 [info] <0.223.0> ra system 'coordination' running pre init for 0 registered servers
2025-11-07 10:57:07.187449+03:00 [info] <0.224.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2025-11-07 10:57:07.197423+03:00 [info] <0.227.0> segment_writer: upgrading segment file names to new format in dirctory /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:07.231941+03:00 [notice] <0.229.0> WAL: ra_coordination_log_wal init, mem-tables table name: ra_coordination_log_open_mem_tables
2025-11-07 10:57:07.258111+03:00 [info] <0.209.0> ra: starting system quorum_queues
2025-11-07 10:57:07.258325+03:00 [info] <0.209.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/quorum/rabbit@rabbit1
2025-11-07 10:57:07.259251+03:00 [info] <0.233.0> ra_log_ets: in system quorum_queues initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,true}]
2025-11-07 10:57:07.261121+03:00 [info] <0.237.0> ra system 'quorum_queues' running pre init for 0 registered servers
2025-11-07 10:57:07.263368+03:00 [info] <0.238.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2025-11-07 10:57:07.264028+03:00 [info] <0.241.0> segment_writer: upgrading segment file names to new format in dirctory /var/lib/rabbitmq/mnesia/rabbit@rabbit1/quorum/rabbit@rabbit1
2025-11-07 10:57:07.267383+03:00 [notice] <0.243.0> WAL: ra_log_wal init, mem-tables table name: ra_log_open_mem_tables
2025-11-07 10:57:07.272896+03:00 [info] <0.245.0> ra_system_recover: ra system 'quorum_queues' server recovery strategy rabbit_quorum_queue:system_recover
2025-11-07 10:57:07.273368+03:00 [info] <0.245.0> [rabbit_quorum_queue:system_recover/1] rabbit not booted, skipping queue recovery
2025-11-07 10:57:07.274078+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:07.274139+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:07.499171+03:00 [notice] <0.247.0> RabbitMQ metadata store: candidate -> leader in term: 1 machine version: 2, last applied 0
2025-11-07 10:57:07.843114+03:00 [info] <0.209.0>
2025-11-07 10:57:07.843114+03:00 [info] <0.209.0>  Starting RabbitMQ 4.2.0 on Erlang 27.3.4.4 [jit]
2025-11-07 10:57:07.843114+03:00 [info] <0.209.0>  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
2025-11-07 10:57:07.843114+03:00 [info] <0.209.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  node           : rabbit@rabbit1
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  home dir       : /var/lib/rabbitmq
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  config file(s) : /etc/rabbitmq/advanced.config
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>                 : /etc/rabbitmq/rabbitmq.conf
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>                 : /etc/rabbitmq/conf.d/10-defaults.conf
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  cookie hash    : etYEGOh1RfQzjA06uBkFIw==
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  log(s)         : /var/log/rabbitmq/rabbit.log
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>                 : <stdout>
2025-11-07 10:57:07.847545+03:00 [info] <0.209.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@rabbit1
2025-11-07 10:57:08.597106+03:00 [info] <0.209.0> Running boot step pre_boot defined by app rabbit
2025-11-07 10:57:08.597343+03:00 [info] <0.209.0> Running boot step rabbit_global_counters defined by app rabbit
2025-11-07 10:57:08.598913+03:00 [info] <0.209.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2025-11-07 10:57:08.599512+03:00 [info] <0.209.0> Running boot step rabbit_core_metrics defined by app rabbit
2025-11-07 10:57:08.601918+03:00 [info] <0.209.0> Running boot step rabbit_alarm defined by app rabbit
2025-11-07 10:57:08.633924+03:00 [info] <0.277.0> Memory high watermark set to 11031 MiB (11566858240 bytes) of 15758 MiB (16524083200 bytes) total
2025-11-07 10:57:08.643864+03:00 [info] <0.279.0> Enabling free disk space monitoring (disk free space: 23047954432, total memory: 16524083200)
2025-11-07 10:57:08.644099+03:00 [info] <0.279.0> Disk free limit set to 50MB
2025-11-07 10:57:08.650634+03:00 [info] <0.209.0> Running boot step code_server_cache defined by app rabbit
2025-11-07 10:57:08.651050+03:00 [info] <0.209.0> Running boot step file_handle_cache defined by app rabbit
2025-11-07 10:57:08.663463+03:00 [info] <0.282.0> Limiting to approx 1048479 file handles (943629 sockets)
2025-11-07 10:57:08.663886+03:00 [info] <0.283.0> FHC read buffering: OFF
2025-11-07 10:57:08.663963+03:00 [info] <0.283.0> FHC write buffering: ON
2025-11-07 10:57:08.664838+03:00 [info] <0.209.0> Running boot step worker_pool defined by app rabbit
2025-11-07 10:57:08.665005+03:00 [info] <0.264.0> Will use 12 processes for default worker pool
2025-11-07 10:57:08.665070+03:00 [info] <0.264.0> Starting worker pool 'worker_pool' with 12 processes in it
2025-11-07 10:57:08.665836+03:00 [info] <0.209.0> Running boot step rabbit_registry defined by app rabbit
2025-11-07 10:57:08.665918+03:00 [info] <0.209.0> Running boot step database defined by app rabbit
2025-11-07 10:57:08.666405+03:00 [info] <0.209.0> Peer discovery: configured backend: rabbit_peer_discovery_classic_config
2025-11-07 10:57:08.667775+03:00 [info] <0.265.0> Feature flags: starting an unclustered node for the first time: all stable feature flags will be enabled by default
2025-11-07 10:57:08.669085+03:00 [notice] <0.265.0> Feature flags: attempt to enable `quorum_queue_non_voters`...
2025-11-07 10:57:08.699505+03:00 [notice] <0.265.0> Feature flags: `quorum_queue_non_voters` enabled
2025-11-07 10:57:08.699926+03:00 [notice] <0.265.0> Feature flags: attempt to enable `detailed_queues_endpoint`...
2025-11-07 10:57:08.728835+03:00 [notice] <0.265.0> Feature flags: `detailed_queues_endpoint` enabled
2025-11-07 10:57:08.729242+03:00 [notice] <0.265.0> Feature flags: attempt to enable `rabbitmq_4.2.0`...
2025-11-07 10:57:08.738461+03:00 [notice] <0.265.0> Feature flags: attempt to enable `rabbitmq_4.1.0`...
2025-11-07 10:57:08.751190+03:00 [notice] <0.265.0> Feature flags: attempt to enable `rabbitmq_4.0.0`...
2025-11-07 10:57:08.778516+03:00 [notice] <0.265.0> Feature flags: `rabbitmq_4.0.0` enabled
2025-11-07 10:57:08.796421+03:00 [notice] <0.265.0> Feature flags: `rabbitmq_4.1.0` enabled
2025-11-07 10:57:08.829526+03:00 [notice] <0.265.0> Feature flags: `rabbitmq_4.2.0` enabled
2025-11-07 10:57:08.829675+03:00 [notice] <0.265.0> Feature flags: attempt to enable `message_containers_deaths_v2`...
2025-11-07 10:57:08.853761+03:00 [notice] <0.265.0> Feature flags: `message_containers_deaths_v2` enabled
2025-11-07 10:57:08.854128+03:00 [notice] <0.265.0> Feature flags: attempt to enable `khepri_db`...
2025-11-07 10:57:08.865427+03:00 [notice] <0.265.0> Feature flags: attempt to enable `ram_node_type`...
2025-11-07 10:57:08.889715+03:00 [notice] <0.265.0> Feature flags: `ram_node_type` enabled
2025-11-07 10:57:08.890012+03:00 [info] <0.341.0> Feature flag `khepri_db`: syncing cluster membership
2025-11-07 10:57:08.920639+03:00 [info] <0.342.0> Syncing Mnesia->Khepri clusters membership
2025-11-07 10:57:08.930912+03:00 [info] <0.341.0> Feature flag `khepri_db`: cluster membership synchronized; members are: ['rabbit@rabbit1']
2025-11-07 10:57:08.931145+03:00 [info] <0.341.0> Feature flag `khepri_db`: unregistering legacy projections
2025-11-07 10:57:08.934456+03:00 [info] <0.341.0> Feature flag `khepri_db`: registering projections
2025-11-07 10:57:11.343004+03:00 [notice] <0.341.0> Feature flags: `khepri_db`: starting migration of 20 tables from Mnesia to Khepri; expect decrease in performance and increase in memory footprint
2025-11-07 10:57:11.343378+03:00 [info] <0.341.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2025-11-07 10:57:11.343527+03:00 [info] <0.341.0> Successfully synced tables from a peer
2025-11-07 10:57:11.343785+03:00 [notice] <0.341.0> Feature flags: `khepri_db`: migration from Mnesia to Khepri finished
2025-11-07 10:57:11.383584+03:00 [notice] <0.45.0> Application mnesia exited with reason: stopped
2025-11-07 10:57:11.385402+03:00 [notice] <0.265.0> Feature flags: `khepri_db` enabled
2025-11-07 10:57:11.385489+03:00 [notice] <0.265.0> Feature flags: attempt to enable `rabbit_exchange_type_local_random`...
2025-11-07 10:57:11.407659+03:00 [notice] <0.265.0> Feature flags: `rabbit_exchange_type_local_random` enabled
2025-11-07 10:57:11.408233+03:00 [info] <0.209.0> DB: virgin node -> run peer discovery
2025-11-07 10:57:12.052508+03:00 [info] <0.209.0> Peer discovery: node 'rabbit@rabbit1' selected for auto-clustering
2025-11-07 10:57:12.052794+03:00 [info] <0.209.0> Peer discovery: will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2025-11-07 10:57:12.087790+03:00 [info] <0.209.0> DB: checking if `rabbit@rabbit1` can join cluster using remote node `rabbit@rabbit1`
2025-11-07 10:57:12.088021+03:00 [notice] <0.209.0> Feature flags: checking nodes `rabbit@rabbit1` and `rabbit@rabbit1` compatibility...
2025-11-07 10:57:12.149858+03:00 [notice] <0.209.0> Feature flags: nodes `rabbit@rabbit1` and `rabbit@rabbit1` are compatible
2025-11-07 10:57:12.359704+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:12.359969+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:12.361127+03:00 [info] <0.391.0> ra_coordination_log_ets: in system coordination initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,false}]
2025-11-07 10:57:12.364962+03:00 [info] <0.395.0> ra system 'coordination' running pre init for 1 registered servers
2025-11-07 10:57:12.367506+03:00 [info] <0.396.0> ra: meta data store initialised for system coordination. 1 record(s) recovered
2025-11-07 10:57:12.368188+03:00 [notice] <0.401.0> WAL: ra_coordination_log_wal init, mem-tables table name: ra_coordination_log_open_mem_tables
2025-11-07 10:57:12.541880+03:00 [notice] <0.406.0> RabbitMQ metadata store: candidate -> leader in term: 2 machine version: 2, last applied 15
2025-11-07 10:57:12.544364+03:00 [warning] <0.406.0> RabbitMQ metadata store command {'$ra_leave',#{ts => 1762502232544,from => {<0.209.0>,[alias|#Ref<0.0.26755.1788408163.2831482883.233997>]}},{rabbitmq_metadata,'rabbit@rabbit1'},after_log_append} NOT appended to log. Reason cluster_change_not_permitted
2025-11-07 10:57:12.747500+03:00 [info] <0.209.0> Deleting server rabbitmq_metadata and its data directory.
2025-11-07 10:57:12.797639+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:12.797985+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:12.798471+03:00 [info] <0.209.0> ra: starting system quorum_queues
2025-11-07 10:57:12.798636+03:00 [info] <0.209.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/quorum/rabbit@rabbit1
2025-11-07 10:57:12.799107+03:00 [info] <0.409.0> ra_log_ets: in system quorum_queues initialising. Mem table opts: [set,{write_concurrency,auto},public,{compressed,true}]
2025-11-07 10:57:12.801666+03:00 [info] <0.413.0> ra system 'quorum_queues' running pre init for 0 registered servers
2025-11-07 10:57:12.804311+03:00 [info] <0.414.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2025-11-07 10:57:12.805094+03:00 [notice] <0.419.0> WAL: ra_log_wal init, mem-tables table name: ra_log_open_mem_tables
2025-11-07 10:57:12.811021+03:00 [info] <0.421.0> ra_system_recover: ra system 'quorum_queues' server recovery strategy rabbit_quorum_queue:system_recover
2025-11-07 10:57:12.811324+03:00 [info] <0.421.0> [rabbit_quorum_queue:system_recover/1] rabbit not booted, skipping queue recovery
2025-11-07 10:57:12.812251+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:12.812326+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:12.838425+03:00 [notice] <0.423.0> RabbitMQ metadata store: candidate -> leader in term: 1 machine version: 2, last applied 0
2025-11-07 10:57:12.839034+03:00 [info] <0.209.0> DB: joining cluster using remote nodes:
2025-11-07 10:57:12.839034+03:00 [info] <0.209.0> ['rabbit@rabbit1','rabbit@rabbit1']
2025-11-07 10:57:12.843916+03:00 [info] <0.209.0> Khepri clustering: Attempt to add node 'rabbit@rabbit1' to cluster ['rabbit@rabbit1','rabbit@rabbit1'] through node 'rabbit@rabbit1'
2025-11-07 10:57:12.844491+03:00 [info] <0.209.0> ra: starting system coordination
2025-11-07 10:57:12.844578+03:00 [info] <0.209.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbit1/coordination/rabbit@rabbit1
2025-11-07 10:57:12.851612+03:00 [info] <0.209.0> Deleting server rabbitmq_metadata and its data directory.
2025-11-07 10:57:12.874820+03:00 [info] <0.209.0> Khepri clustering: Node 'rabbit@rabbit1' successfully added to cluster "rabbitmq_metadata" through node 'rabbit@rabbit1'
2025-11-07 10:57:12.939825+03:00 [info] <0.425.0> RabbitMQ metadata store: detected a new leader {rabbitmq_metadata,'rabbit@rabbit1'} in term 5
2025-11-07 10:57:12.951164+03:00 [info] <0.209.0> Peer discovery: this node (rabbit@rabbit1) successfully joined node 'rabbit@rabbit1' cluster
2025-11-07 10:57:12.951460+03:00 [info] <0.209.0> Peer discovery: will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0> Found the following metadata store members: [{rabbitmq_metadata,
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0>                                               'rabbit@rabbit1'},
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0>                                              {rabbitmq_metadata,
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0>                                               'rabbit@rabbit1'},
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0>                                              {rabbitmq_metadata,
2025-11-07 10:57:13.107379+03:00 [notice] <0.209.0>                                               'rabbit@rabbit1'}]
2025-11-07 10:57:13.164943+03:00 [info] <0.209.0> Running boot step tracking_metadata_store defined by app rabbit
2025-11-07 10:57:13.165453+03:00 [info] <0.428.0> Setting up a table for connection tracking on this node: tracked_connection
2025-11-07 10:57:13.165726+03:00 [info] <0.428.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2025-11-07 10:57:13.166063+03:00 [info] <0.428.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2025-11-07 10:57:13.166363+03:00 [info] <0.428.0> Setting up a table for channel tracking on this node: tracked_channel
2025-11-07 10:57:13.166500+03:00 [info] <0.428.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2025-11-07 10:57:13.166950+03:00 [info] <0.209.0> Running boot step networking_metadata_store defined by app rabbit
2025-11-07 10:57:13.167391+03:00 [info] <0.209.0> Running boot step feature_flags defined by app rabbit
2025-11-07 10:57:13.168012+03:00 [info] <0.209.0> Running boot step auth_backend_plugins_check defined by app rabbit
2025-11-07 10:57:13.468236+03:00 [info] <0.209.0> Running boot step codec_correctness_check defined by app rabbit
2025-11-07 10:57:13.468399+03:00 [info] <0.209.0> Running boot step external_infrastructure defined by app rabbit
2025-11-07 10:57:13.468550+03:00 [info] <0.209.0> Running boot step rabbit_event defined by app rabbit
2025-11-07 10:57:13.468972+03:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2025-11-07 10:57:13.469191+03:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_anonymous defined by app rabbit
2025-11-07 10:57:13.469583+03:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2025-11-07 10:57:13.470053+03:00 [info] <0.209.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2025-11-07 10:57:13.470215+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2025-11-07 10:57:13.470358+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2025-11-07 10:57:13.470480+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2025-11-07 10:57:13.470651+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_local_random defined by app rabbit
2025-11-07 10:57:13.470822+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2025-11-07 10:57:13.470949+03:00 [info] <0.209.0> Running boot step rabbit_priority_queue defined by app rabbit
2025-11-07 10:57:13.471109+03:00 [info] <0.209.0> Priority queues enabled, real BQ is rabbit_variable_queue
2025-11-07 10:57:13.471208+03:00 [info] <0.209.0> Running boot step kernel_ready defined by app rabbit
2025-11-07 10:57:13.471266+03:00 [info] <0.209.0> Running boot step ldap_pool defined by app rabbitmq_auth_backend_ldap
2025-11-07 10:57:13.471366+03:00 [info] <0.264.0> Starting worker pool 'ldap_pool' with 64 processes in it
2025-11-07 10:57:13.474793+03:00 [info] <0.209.0> Running boot step pg_local_amqp_connection defined by app rabbit
2025-11-07 10:57:13.489260+03:00 [info] <0.209.0> Running boot step pg_local_amqp_session defined by app rabbit
2025-11-07 10:57:13.489689+03:00 [info] <0.209.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2025-11-07 10:57:13.490227+03:00 [info] <0.209.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2025-11-07 10:57:13.492600+03:00 [info] <0.515.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2025-11-07 10:57:13.492825+03:00 [info] <0.209.0> Running boot step guid_generator defined by app rabbit
2025-11-07 10:57:13.496784+03:00 [info] <0.209.0> Running boot step rabbit_node_monitor defined by app rabbit
2025-11-07 10:57:13.497170+03:00 [info] <0.519.0> Starting rabbit_node_monitor (partition handling strategy unapplicable with Khepri)
2025-11-07 10:57:13.497637+03:00 [info] <0.209.0> Running boot step delegate_sup defined by app rabbit
2025-11-07 10:57:13.498259+03:00 [info] <0.209.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2025-11-07 10:57:13.498371+03:00 [info] <0.209.0> Running boot step core_initialized defined by app rabbit
2025-11-07 10:57:13.498398+03:00 [info] <0.209.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2025-11-07 10:57:13.498456+03:00 [info] <0.209.0> Running boot step rabbit_classic_queue_type defined by app rabbit
2025-11-07 10:57:13.498549+03:00 [info] <0.209.0> Running boot step rabbit_classic_queue defined by app rabbit
2025-11-07 10:57:13.498652+03:00 [info] <0.209.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2025-11-07 10:57:13.498700+03:00 [info] <0.209.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2025-11-07 10:57:13.498840+03:00 [info] <0.209.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2025-11-07 10:57:13.504555+03:00 [info] <0.209.0> Running boot step rabbit_policies defined by app rabbit
2025-11-07 10:57:13.505911+03:00 [info] <0.209.0> Running boot step rabbit_policy defined by app rabbit
2025-11-07 10:57:13.506210+03:00 [info] <0.209.0> Running boot step rabbit_quorum_event_subscriber defined by app rabbit
2025-11-07 10:57:13.506425+03:00 [info] <0.209.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2025-11-07 10:57:13.506549+03:00 [info] <0.209.0> Running boot step rabbit_quorum_queue defined by app rabbit
2025-11-07 10:57:13.506823+03:00 [info] <0.209.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2025-11-07 10:57:13.507258+03:00 [info] <0.209.0> Running boot step rabbit_vhost_limit defined by app rabbit
2025-11-07 10:57:13.507374+03:00 [info] <0.209.0> Running boot step rabbit_exchange_type_delayed_message defined by app rabbitmq_delayed_message_exchange
2025-11-07 10:57:13.507483+03:00 [info] <0.209.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2025-11-07 10:57:13.507574+03:00 [info] <0.209.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2025-11-07 10:57:13.507700+03:00 [info] <0.209.0> Management plugin: using rates mode 'basic'
2025-11-07 10:57:13.518758+03:00 [info] <0.209.0> Running boot step recovery defined by app rabbit
2025-11-07 10:57:13.525444+03:00 [info] <0.545.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbit1/msg_stores/vhosts/62LCMMKRHOU04O0GRYM4NUNP8' for vhost '/' exists
2025-11-07 10:57:13.527749+03:00 [info] <0.545.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
2025-11-07 10:57:13.550702+03:00 [info] <0.545.0> Starting message stores for vhost '/'
2025-11-07 10:57:13.554323+03:00 [info] <0.545.0> Started message store of type transient for vhost '/'
2025-11-07 10:57:13.614950+03:00 [warning] <0.561.0> Message store "62LCMMKRHOU04O0GRYM4NUNP8/msg_store_persistent": rebuilding indices from scratch
2025-11-07 10:57:13.617446+03:00 [info] <0.545.0> Started message store of type persistent for vhost '/'
2025-11-07 10:57:13.617765+03:00 [info] <0.545.0> Recovering 0 queues of type rabbit_classic_queue took 89ms

2025-11-07 10:57:13.791007+03:00 [info] <0.596.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbit1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2025-11-07 10:57:13.792611+03:00 [info] <0.596.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
2025-11-07 10:57:13.801776+03:00 [info] <0.596.0> Starting message stores for vhost '/'
2025-11-07 10:57:13.805434+03:00 [info] <0.596.0> Started message store of type transient for vhost '/'
2025-11-07 10:57:13.806978+03:00 [warning] <0.607.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2025-11-07 10:57:13.808270+03:00 [info] <0.596.0> Started message store of type persistent for vhost '/'
2025-11-07 10:57:13.808474+03:00 [info] <0.596.0> Recovering 0 queues of type rabbit_classic_queue took 14ms
2025-11-07 10:57:13.815673+03:00 [info] <0.209.0> Running boot step prevent_startup_if_node_was_reset defined by app rabbit
2025-11-07 10:57:13.815933+03:00 [info] <0.209.0> Running boot step empty_db_check defined by app rabbit
2025-11-07 10:57:13.816675+03:00 [info] <0.209.0> Will not seed default virtual host and user: have definitions to load...
2025-11-07 10:57:13.816791+03:00 [info] <0.209.0> Running boot step rabbit_observer_cli defined by app rabbit
2025-11-07 10:57:13.816905+03:00 [info] <0.209.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2025-11-07 10:57:13.817110+03:00 [info] <0.209.0> Running boot step background_gc defined by app rabbit
2025-11-07 10:57:13.817277+03:00 [info] <0.209.0> Running boot step rabbit_observer_cli_classic_queues defined by app rabbit
2025-11-07 10:57:13.817331+03:00 [info] <0.209.0> Running boot step rabbit_observer_cli_quorum_queues defined by app rabbit
2025-11-07 10:57:13.817527+03:00 [info] <0.209.0> Running boot step routing_ready defined by app rabbit
2025-11-07 10:57:13.817613+03:00 [info] <0.209.0> Running boot step pre_flight defined by app rabbit
2025-11-07 10:57:13.817712+03:00 [info] <0.209.0> Running boot step notify_cluster defined by app rabbit
2025-11-07 10:57:13.817882+03:00 [info] <0.519.0> rabbit on node 'rabbit@rabbit1' up
2025-11-07 10:57:13.817831+03:00 [info] <0.209.0> Running boot step networking defined by app rabbit
2025-11-07 10:57:13.818026+03:00 [info] <0.209.0> Running boot step rabbit_delayed_message defined by app rabbitmq_delayed_message_exchange
2025-11-07 10:57:13.818474+03:00 [info] <0.519.0> rabbit on node 'rabbit@rabbit1' up
2025-11-07 10:57:13.913888+03:00 [info] <0.209.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2025-11-07 10:57:13.914091+03:00 [info] <0.209.0> Successfully synced tables from a peer
2025-11-07 10:57:13.914129+03:00 [info] <0.209.0> Running boot step rabbit_quorum_queue_periodic_membership_reconciliation defined by app rabbit
2025-11-07 10:57:13.914495+03:00 [info] <0.209.0> Running boot step definition_import_worker_pool defined by app rabbit
2025-11-07 10:57:13.914553+03:00 [info] <0.264.0> Starting worker pool 'definition_import_pool' with 12 processes in it
2025-11-07 10:57:13.915804+03:00 [info] <0.209.0> Running boot step cluster_name defined by app rabbit
2025-11-07 10:57:13.915883+03:00 [info] <0.209.0> Running boot step virtual_host_reconciliation defined by app rabbit
2025-11-07 10:57:13.923291+03:00 [info] <0.209.0> Running boot step direct_client defined by app rabbit
2025-11-07 10:57:13.923567+03:00 [info] <0.209.0> Running boot step cluster_tags defined by app rabbit
2025-11-07 10:57:13.928562+03:00 [info] <0.209.0> Running boot step rabbit_quorum_queue_type defined by app rabbit
2025-11-07 10:57:13.929138+03:00 [info] <0.209.0> Running boot step rabbit_stream_queue defined by app rabbit
2025-11-07 10:57:13.929367+03:00 [info] <0.209.0> Running boot step rabbit_delayed_message_supervisor defined by app rabbitmq_delayed_message_exchange
2025-11-07 10:57:13.930276+03:00 [info] <0.209.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2025-11-07 10:57:13.930547+03:00 [info] <0.696.0> Resetting node maintenance status
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0> Deprecated features: `management_metrics_collection`: Feature `management_metrics_collection` is deprecated.
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0> By default, this feature can still be used for now.
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0> Its use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0> To continue using this feature when it is not permitted by default, set the following parameter in your configuration:
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0>     "deprecated_features.permit.management_metrics_collection = true"
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0> To test RabbitMQ as if the feature was removed, set this in your configuration:
2025-11-07 10:57:14.294462+03:00 [warning] <0.718.0>     "deprecated_features.permit.management_metrics_collection = false"
2025-11-07 10:57:14.608886+03:00 [info] <0.754.0> Management plugin: HTTPS listener started on port 15671
2025-11-07 10:57:14.609346+03:00 [info] <0.784.0> Statistics database started.
2025-11-07 10:57:14.609444+03:00 [info] <0.783.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2025-11-07 10:57:14.609861+03:00 [info] <0.713.0> opening log file: "/var/log/rabbitmq/access/access.log.2025_11_07_07"
2025-11-07 10:57:14.669475+03:00 [info] <0.805.0> Prometheus metrics: HTTPS listener started on port 15691
2025-11-07 10:57:14.669808+03:00 [info] <0.696.0> Ready to start client connection listeners
2025-11-07 10:57:14.675358+03:00 [info] <0.853.0> started TLS (SSL) listener on [::]:5671
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0> Server startup complete; 7 plugins started.
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_prometheus
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_auth_backend_ldap
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_top
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_delayed_message_exchange
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_management
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_management_agent
2025-11-07 10:57:14.868123+03:00 [info] <0.696.0>  * rabbitmq_web_dispatch
2025-11-07 10:57:14.936047+03:00 [info] <0.10.0> Time to start RabbitMQ: 14234 ms
2025-11-07 10:57:16.296300+03:00 [debug] <0.865.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:21.068537+03:00 [debug] <0.967.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:22.865891+03:00 [debug] <0.1011.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:26.052072+03:00 [debug] <0.1074.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:27.834595+03:00 [debug] <0.1091.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:31.093336+03:00 [debug] <0.1144.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:32.894459+03:00 [debug] <0.1161.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:36.053134+03:00 [debug] <0.1207.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:37.840774+03:00 [debug] <0.1224.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:41.048824+03:00 [debug] <0.1273.0> error when receiving proxy header: TCP socket was closed prematurely
2025-11-07 10:57:42.861178+03:00 [debug] <0.1290.0> error when receiving proxy header: TCP socket was closed prematurely

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

listeners.ssl.default = 5671
loopback_users.guest = false
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.cacertfile=/etc/rabbitmq/certificates/cacert.pem
ssl_options.certfile=/etc/rabbitmq/certificates/cert.pem
ssl_options.keyfile=/etc/rabbitmq/certificates/key.pem


prometheus.ssl.port       = 15691
prometheus.ssl.cacertfile = /etc/rabbitmq/certificates/cacert.pem
prometheus.ssl.certfile   = /etc/rabbitmq/certificates/cert.pem
prometheus.ssl.keyfile    = /etc/rabbitmq/certificates/key.pem

ssl_options.versions.1 = tlsv1.2

proxy_protocol=true

management.ssl.port       = 15671
management.ssl.cacertfile = /etc/rabbitmq/certificates/cacert.pem
management.ssl.certfile   = /etc/rabbitmq/certificates/cert.pem
management.ssl.keyfile    = /etc/rabbitmq/certificates/key.pem
management.http_log_dir = /var/log/rabbitmq/access


cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@rabbit1
cluster_formation.classic_config.nodes.2 = rabbit@rabbit2
cluster_formation.classic_config.nodes.3 = rabbit@rabbit3

auth_ldap.ssl_options.verify = verify_none

log.file = rabbit.log
log.dir = /var/log/rabbitmq
log.file.level = debug
log.connection.level = debug
log.channel.level = debug
log.queue.level = debug
log.mirroring.level = error

log.file.rotation.date = $D0
log.file.rotation.count = 30


vm_memory_high_watermark.relative = 0.7

listeners.tcp = none
prometheus.tcp.listener = none

Steps to deploy RabbitMQ cluster

I deploy as docker containers.

Steps to reproduce the behavior in question

I deploy as a docker container my cluster.

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

I try to upgrade rabbitMQ 4.1.0 to 4.2.0, i could upgrade version but there is TCP errors after upgrade in the log file . And error keeps coming.

"error when receiving proxy header: TCP socket was closed prematurely"

I am using "proxy_protocol=true" in config file. But it is okay 4.1.0 version. Are there any changes about proxy protocol policies in new version 4.2.0?

[michaelklishin]

@skry-dev all meaningful changes are documented in the release notes.

We havne't had any Proxy Protocol-related changes in a few years.

"There are TCP errors" is not something we can comment on, we do not guess in this community.