Observed memory leak in ssl library: Python 3.14 GC issue

[HenriBlacksmith]

Bug report

Bug description:

I already opened a bug on urllib3 for this issue, it looks that may application using MSAL for Python library (which relies on requests , which relies on urllib3, which relies on ssl) is leaking memory when being upgraded from Python 3.13 to Python 3.14 (I tested 3.14.{0, 1, 2})

The details are explained in the urllib3 ticket including outputs from memray and it points to ssl package (precisely ssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-142516: Memory leak in SSLContext #143585
• gh-142516: fix reference leaks in ssl.SSLContext objects #143685
• [3.14] gh-142516: fix reference leaks in ssl.SSLContext objects (GH-143685) #145075
• [3.13] gh-142516: fix reference leaks in ssl.SSLContext objects (GH-143685) (GH-145075) #148371
• gh-142516: Forward-port generational GC #148687
• [3.14] Forward-port generational GC (GH-142516) #148694

[vstinner]

The details are explained in the urllib3 ticket including outputs from memray and it points to ssl package (precisely ssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738

Please provide a reproducer, if possible using only the ssl module.

[zweizeichen]

No reproducer but if it helps: very similar situation here when connecting to valkey with a custom CA cert.

[zweizeichen]

I did some more testing to come up with a minimal repro: seems like rss just keeps growing while the heap stays somewhat constant. I was unable to reproduce it using a simple sync case locally and on the cluster and this started to smell an awful lot like #109534 which might be async-related. Like @HenriBlacksmith's case we're also seeing this in an ASGI app on k8s using the debian trixie base image since the 3.14 release.

I'll attempt to redeploy this glibc-less and report back.

Update: the alleged leak is still there even in an alpine container

[zweizeichen]

Potentially related: #84904

[HenriBlacksmith]

I applied a fix that improved the memory situation on my other FastAPI app by reducing the calls generated by MSAL.

I applied a few dependency upgrades on top including a bump of the Python image from 3.14.0-slim to 3.14.2-slim and it caused the memory leak to be worse.

Though I also need to collect detailed figures about this one as urllib3 version is different in this app because of some version constraints on a library.

I will try to get more info from memray on this one.

[HenriBlacksmith]

I was able to export new memray profiles from my other application.

This time the leak is coming from httpx which also relies on ssl package under the hood:

Sorry for the cuts on the capture, I am trying to "anonymize" things

[zweizeichen]

Thank you! I'm not 100% sure how to interpret these memray outputs to be honest, because if I understand this correctly this shows us where allocations do happen but not necessarily where the leak is (it may be correlated though). This is useful for finding out 'which objects eat up my memory?'. Technically speaking memray will only show us the allocations/leaks the python allocator knows about, but the leak kind of escapes the realm of the allocator by definition. Another layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.

I'm going to try to find some time to give the minimal (hopefully stdlib only) repro another go and explore analyzing a full heap dump.

[HenriBlacksmith]

I am trying to use valgrind instead of memray to get more details (but I am a bit out of my comfort zone to be honest), it does not report much until k8s restarts the container.

The command looks like:

valgrind --tool=memcheck --log-file=/tmp/valgrind.log --trace-children=yes --leak-check=yes --verbose
              python -m uvicorn app.main:shell --host 0.0.0.0 --log-config logging.yaml

But nothing really comes out of it

[HenriBlacksmith]

Another layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.

I just built an Alpine container and ran it and the behavior is similar (based on python:3.14.2-alpine3.23)

I am running out of ideas, if anyone has guidance to help with investigating this, I would be really happy to test those, if I have extra time, I will try to build a MRO but I am not sure how it can be done

[HenriBlacksmith]

Good news, I might have a MRE, it is using httpx but that's the best I was able to do, I will share the whole code in a repo, including commands to export the graphs.

Here is the code:

import httpx
import asyncio

async def _main() -> None:
    while(1):
        async with httpx.AsyncClient() as client:
            await client.request("GET", "https://example.com/")
        await asyncio.sleep(1)

if __name__ == "__main__":
    asyncio.run(_main())

FROM astral/uv:0.9.17-python3.14-bookworm AS builder
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=0

# Create and set the working directory
WORKDIR /mro-app

# Install runtime dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --frozen --no-install-project --no-dev

# Copy the application code (excluding files in .dockerignore)
COPY . .
RUN --mount=type=cache,target=/root/.cache/uv \
        uv sync --frozen --no-dev

# Use an official lightweight Python image with slim variant
FROM python:3.14.2-slim

# Set environment variables
# See: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED
ENV PYTHONUNBUFFERED=1

# Create and set the working directory
WORKDIR /mro-app

# Expose the port the app runs on
EXPOSE 8000

# Copy the application from the builder
COPY --from=builder /mro-app /mro-app

# Place executables in the environment at the front of the path
ENV PATH="/mro-app/.venv/bin:$PATH"

# Define the default command to run when the container starts
CMD ["memray", "run", "--native", "--output", "/tmp/capture.bin", "main.py"]

[project]
name = "mro"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.14"
dependencies = [
    "httpx>=0.28.1",
    "memray>=1.19.1",
    "urllib3>=2.6.2",
]

etc. will build a repo asap with all the other files

[HenriBlacksmith]

Here is the repo, I hope it is an actual valid MRE and not just "normal" usage: https://github.com/HenriBlacksmith/python-ssl-memory-usage-mro/tree/main

[HenriBlacksmith]

And I was finally able to get the native symbols:

I will try to compare with older Python versions to see if the sensitivity to Python version holds with the MRE.

I was able to reproduce on my Mac in a non-Docker environment.