Federation sender memory leak in synapse 1.49.1

[jkhsjdhjs]

Description

When running synapse with federation sender workers, the workers start to use more and more memory over time, around 360MB per worker per day. I use 2 federation senders, so for me memory usage increased by 720MB per day. Eventually the LXC I'm running my synapse setup in reached its memory limit, memory pressure was applied, slowing down and stalling everything.

The following graph shows the memory usage of my synapse LXC. Restarts occured on 2026-03-07 and on 2026-03-15. Additionally, synapse was upgraded on the latter date, from 1.144.0 to 1.149.1.
The graph shows that with 1.144.0 memory usage was fairly stable, while it constantly increases with 1.149.1.

The following htop screenshot shows that the high memory usage is limited to the federation sender processes:

Steps to reproduce

Run synapse with federation senders

Homeserver

own homeserver running synapse 1.149.1

Synapse Version

1.149.1

Installation Method

Other (please mention below)

Database

PostgreSQL 18.3

Workers

Multiple workers

Platform

Arch Linux, LXC on Proxmox, Ryzen 5950X, LXC Limited to 20GiB of memory
Synapse installed via distro packages

Configuration

I'm using synapse with a main process and 3 workers. 1 generic_worker and 2 federation senders. Incoming requests are only handled by the generic_worker and main process based on static rules. No load balancing is done.

Furthermore, I'm using these experimental features:

experimental_features:
  # MSC3266: Room summary API. Used for knocking over federation
  msc3266_enabled: true
  # MSC4222 needed for syncv2 state_after. This allow clients to
  # correctly track the state of the room.
  msc4222_enabled: true

Relevant log output

Only error I see is

synapse.storage.databases.main.event_federation: [GET-3480033] Unexpectedly found that events don't have chain IDs in room !iMZEhwCvbfeAYUxAjZ:t2l.io: {'$ifBl2jg9HL3gG2y3mTDrlOwXzbEb3EcMRIH9hP6ghpM', '$IAIeX7zg7JGjy8KQASeYApVxmArz_54>

But I have been seeing this error before the upgrade to 1.149.1 as well. On 1.440.0 I didn't see any memory leak.

Anything else that would be useful to know?

@JacksonChen666 also experiences the same issue on 1.149.1.
1.150.0 is not available via my distributions packages yet, but the changelog also doesn't indicate any fixes regarding memory leaks.

[JacksonChen666]

Hi, I've been dealing this on jacksonchen666.com. I've been dealing with this for at least a few weeks now, as I'm still testing various changes because memory leaks take a week for it to cause my server be starved of memory.

I have tons of notes. First, setup:

• Synapse PyPI 1.147.1 and 1.149.1
• PostgreSQL
• Fedora Linux 43
• Multiple workers:
  • A single outbound federation worker*
    • * This was accidentally handling /sync endpoints, and one of the sync workers was handling nothing, due to a configuration error in the ports setup. This was fixed on 2026-03-30, and seems to probably been in place since 2024-11-01 with no major breaking issues.
  • 2 /sync workers
  • A main process
• Synapse modules:
  • https://github.com/maunium/synapse-http-antispam with Draupnir (I am not sure if I can confirm this actually works. I'm not sure if I've ever tested? But it has existed for a while)
• Presence is enabled.
• Cache auto tuning was enabled until disabled on 2026-03-30
• System has 12 GiB of RAM, 12 GiB of swap, and a lot of other things being hosted at the same time as well.

I'm just gonna go over the most notable things from my perspective:

• Memory and swap usage increases in steps, roughly. I think.
• Swap usage is usually multiple gigabytes large, like 6 GiB, before like I run out of memory.
• It's always federator1 using the most amount of memory. Main processes and syncers aren't notable. They truly aren't. (300 MiB total in RAM, 300 MiB total in swap on 2026-03-30)
• Open FDs for federator1 look like it's leaking (when idle, there's always a minimum amount, slowly increasing over the weeks)
  • 1.135.2 did not have this symptom, but my Prometheus data is very limited before the 1.147.1 upgrade (I ran into my Prometheus storage retention), but it's still pretty obvious.
• I don't have concrete reproduction steps to make memory usage go up.

Issue and setup history:

• 2026-02-17: I was on Synapse 1.135.2 until I upgraded to Fedora 43, where I was forced to reinstall Synapse (and implicitly upgrade) because of a Python version upgrade. Now 1.147.1.
• 2026-03-06: Memory nearly exhausted event, full host restart.
• 2026-03-12: Memory nearly exhausted event, restarted federator1, then had to do a full host restart anyways due to unrelated DNS issues
  • Upgrade from 1.147.1 to 1.149.1
• 2026-03-18: OOM triggered, killed the federator1 process. Did a full host restart due to OOM triggering.
  • Increased Synapse log retention from 1 day to 14 days
• 2026-03-21: Restarted all(?) of Synapse pre-emptively due to federator1 memory usage.
  • Properly setup jemalloc (actually referece a jemalloc library thing). Probably incorrectly setup since I setup Synapse (2021)
• 2026-03-27: Memory nearly exhausted event, took a while to restart federator1, to the point where systemd just killed it instead.
  • 2026-03-30: Fix slightly incorrect workers setup, where I was making the outbound federation worker handle /sync.
  • 2026-03-30: Also disabled cache auto tuning.

This might not be all of the details. I have preserved the following information:

• All available prometheus data until 2026-03-30
• atop history since 2026-02-14
• Synapse logs from 2026-03-17 to 2026-03-30 (until the point where I archived logs)

This covers most things, but probably not everything. Ask me for more info if wanted, especially logs (which are giant).

[JacksonChen666]

@jkhsjdhjs you should also probably mention what caching settings you have. You mentioned it in the Synapse admin room, but not here.

[jkhsjdhjs]

Yep, good point! I'm not using any special cache settings, everything regarding caching is commented out in my homeserver.yaml:

## Caching ##

# Caching can be configured through the following options.
#
# A cache 'factor' is a multiplier that can be applied to each of
# Synapse's caches in order to increase or decrease the maximum
# number of entries that can be stored.

# The number of events to cache in memory. Not affected by
# caches.global_factor.
#
#event_cache_size: 10K

caches:
   # Controls the global cache factor, which is the default cache factor
   # for all caches if a specific factor for that cache is not otherwise
   # set.
   #
   # This can also be set by the "SYNAPSE_CACHE_FACTOR" environment
   # variable. Setting by environment variable takes priority over
   # setting through the config file.
   #
   # Defaults to 0.5, which will half the size of all caches.
   #
   #global_factor: 1.0

   # A dictionary of cache name to cache factor for that individual
   # cache. Overrides the global cache factor for a given cache.
   #
   # These can also be set through environment variables comprised
   # of "SYNAPSE_CACHE_FACTOR_" + the name of the cache in capital
   # letters and underscores. Setting by environment variable
   # takes priority over setting through the config file.
   # Ex. SYNAPSE_CACHE_FACTOR_GET_USERS_WHO_SHARE_ROOM_WITH_USER=2.0
   #
   # Some caches have '*' and other characters that are not
   # alphanumeric or underscores. These caches can be named with or
   # without the special characters stripped. For example, to specify
   # the cache factor for `*stateGroupCache*` via an environment
   # variable would be `SYNAPSE_CACHE_FACTOR_STATEGROUPCACHE=2.0`.
   #
   per_cache_factors:
     #get_users_who_share_room_with_user: 2.0

[JacksonChen666]

I've noticed in Grafana that "In-memory federation transmission queues" has a bunch of pending EDUs (like couple hundred, when calm), with open FDs also correlating with that stat, and I guess all of those roughly correlating with memory usage as well.

The pending EDU count starts from 0 on startup, then every time a bunch of EDUs gets federated (mostly m.presence events), there will be +1 pending EDU that seems to generally never go away. Sometimes it manages to decrease, but most of the time, every EDU federation wave (mostly from m.presence) adds 1 pending EDU to the pile. Piling up at least a couple hundred EDUs take a real while though, at least 6 hours of me using matrix on my phone to get a couple hundred. I am not sure which homeserver is responsible for this mess.

I also tried version bisecting, downgraded to 1.147.0 (although with a deeper downgrade in between) and ran into all clients (nheko, Draupnir, SchildiNext? and probably more) encountering Invalid stream token errors on /sync, which means I just can't do it (even though the documentation supposedly says it's possible). Upgrading back up to 1.149.1 is fine. So pinning down via downgrading is seemingly not an option for me. Anyone else trying to narrow down in a dev environment or something can take this versions list as a reference:

v1.144.0 (last good by jkhsjdhjs)
v1.145.0
v1.146.0
v1.147.0
v1.147.1 (first bad by JacksonChen666)

[kroeckx]

I see the same issue with 1.146.0. It was fine until some time ago.

[genewitch]

this started when i upgraded to 1.151, and continues in 1.152

it crashes after running for a little over a day, due to OOM. you can plainly see it used to not run out:

there is no indication from synapse that anything bad is happening.

note: i don't run the microservices version, i have sqlite3 homeserver.db and redis, and coturn on this machine, for a half decade. so i do a pip upgrade and it upgrades everything. My point is, at least one person above is running "federator" and whatnot, i don't.

hope this helps.

[JacksonChen666]

@jkhsjdhjs @kroeckx @genewitch Hey, can all of you show your open FDs graph from the Grafana dashboard, if you have metrics setup for Synapse?

I've found a workaround, and it seems to be disabling presence (I did it on 2026-05-01), which reduced the lingering open FDs (and memory usage). It's a workaround though, not a real fix for the issue, but it would be interesting to see if all of you are seeing similar things.

(It's not about the big spikes, it's the minimum open FDs creeping up, see the range within less than 500)

[genewitch]

@JacksonChen666

i have it logging open fd for python with 1 second granularity.
watch -n 1 'ls -l /proc/18332/fd/ |wc -l >>/tmp/openfd.txt'
then

awk 'NR%300==1 {if(NR>1) print min; min=$1}
     NR%300!=1 {if($1<min) min=$1}
     END {print min}' /tmp/openfd.txt > /tmp/openfd_baseline.txt

Graphs are made with a python script

import matplotlib.pyplot as plt
import numpy as np

# Load just the baseline minima
with open('openfd_baseline.txt') as f:
    minima = [int(line.strip()) for line in f if line.strip()]

# Create time axis (5-minute blocks)
plt.figure(figsize=(12, 6))
plt.plot(range(len(minima)), minima, 'b-o', linewidth=2, markersize=6)
plt.ylabel('Minimum Open FDs')
plt.xlabel('5-minute blocks')
plt.title('FD Baseline Trend (Spikes Removed)')
plt.grid(True, alpha=0.3)

# Trend line
x = np.arange(len(minima))
z = np.polyfit(x, minima, 1)
p = np.poly1d(z)
plt.plot(x, p(x), "r--", alpha=0.7, label=f'Trend: {z[0]:.2f} FDs/block')
plt.legend()

plt.show()

print(f"Baseline sequence: {minima}")
print(f"Change: {minima[-1] - minima[0]} FDs")

13 hours open fd min reporting:

as the image says, since starting logging, the trend is 0.26 more open FD per five minutes. and for 90% of that time, no one was using the server; it was just doing it's federation with the handful of chatrooms our 3 users are in

edit after crash: just hit OOM after burning through swap and the python process is hung.

raw:

i should note that this constitutes a new open FD every 15 minutes, and on this particular machine with 8GB of ram, the memory usage of python goes up by 1% every 15 minutes as well, until i run out (18-24 hours after synctl start)

[genewitch]

I am running synapse via memray so i can get more detailed allocation statistics; unfortunately i don't have debug symbols for python. I'm approaching the limits of my understanding of what data i am generating, so if i should stop, someone let me know.

Since i am running using memray run --native, it isn't using pymalloc, and the memory doesn't appear to be leaking. This could be informational by itself. again, no pymalloc, memory holding at 2.5% of 8GB

anyhow the memray stats

🥇 Top 5 largest allocating locations (by size):
        - bio_read:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:2026 -> 3.506GB
        - doRead:/home/genewitch/synapse/lib/python3.10/site-packages/twisted/internet/tcp.py:241 -> 3.193GB
        - do_handshake:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:2074 -> 3.042GB
        - recv:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:1944 -> 2.069GB
        - recv:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:1948 -> 1.333GB

🥇 Top 5 largest allocating locations (by number of allocations):
        - do_handshake:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:2074 -> 21425808
        - _do_execute:/home/genewitch/synapse/lib/python3.10/site-packages/synapse/storage/database.py:520 -> 1337791
        - writeCollectionEnd:/home/genewitch/synapse/lib/python3.10/site-packages/thrift/protocol/TCompactProtocol.py:246 -> 1123642
        - _runCallbacks:/home/genewitch/synapse/lib/python3.10/site-packages/twisted/internet/defer.py:857 -> 915879
        - __init__:/home/genewitch/synapse/lib/python3.10/site-packages/OpenSSL/SSL.py:717 -> 733460

[genewitch]

All i can say is, with the command memray run --native --output synapse_trace.bin /home/genewitch/synapse/bin/run_synapse.py --config-path /home/genewitch/synapse/homeserver.yaml
and run_synapse.py:

#!/home/genewitch/synapse/bin/python
import sys
from synapse.app.homeserver import main

if __name__ == '__main__':
    sys.argv[0] = 'synapse.app.homeserver'
    sys.exit(main())
run_synapse.py lines 1-7/7 (END)

i haven't seen a memory leak. I think this means the issue is with synapse and pymalloc, as just attaching memray to a running synapse until it crashed had the report say it was using pymalloc.

someone who is smarter and more conversant in this will hopefully understand my grunts.

thanks, all

[genewitch]

i am going to re-iterate that i can reproduce a memory leak, but i have no way to determine what is leaking memory, other than my minimum open FD creeps up.

When i attach memray in such a way as to capture actual allocations and de-allocations (--native), no leak.

note: leak occurs without memray causing OOM within 24 hours on my server; versions > 1.50.x are all i've tested. In am not updating until someone marks this issue as "closed"

[mtippmann]

i am going to re-iterate that i can reproduce a memory leak, but i have no way to determine what is leaking memory, other than my minimum open FD creeps up.

When i attach memray in such a way as to capture actual allocations and de-allocations (--native), no leak.

note: leak occurs without memray causing OOM within 24 hours on my server; versions > 1.50.x are all i've tested. In am not updating until someone marks this issue as "closed"

what python are you on? I've had a big memory leak running 3.14.3 via pyenv and it looks like i've hit python/cpython#142516 which was fixed in 3.14.5 - this closely matches your data (i.e. openssl is on the top)

looks like it's unrelated to synapse itself

[jkhsjdhjs]

@mtippmann Thanks a lot, that's very valuable information and sounds like it may be the problem for me as well. Until now, I was running synapse 1.150.0 on python 3.14.4 and was still experiencing the issue, although the memory ramped up slowly for me:

I have just updated to synapse 1.151.0 and python 3.14.5, I am hopeful that this will solve the problem.

[kroeckx]

I'm running on Debian stable, which is python 3.13.5. I think the 3.13 branch is affected as well, and the fix has been commited in the 3.13 branch too a few days ago, so not in a release yet.

[jkhsjdhjs]

I have been running python 3.14.5 for a week, seems to be fixed now: