Skip to content

[3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA (GH-156003) (GH-156362) - #156738

Draft
miss-islington wants to merge 2 commits into
python:3.13from
miss-islington:backport-1b424c0-3.13
Draft

[3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA (GH-156003) (GH-156362)#156738
miss-islington wants to merge 2 commits into
python:3.13from
miss-islington:backport-1b424c0-3.13

Conversation

@miss-islington

@miss-islington miss-islington commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2 and LZMA,
members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin encukou@gmail.com
Co-authored-by: tonghuaroot tonghuaroot@gmail.com

…tandard (pythonGH-156003) (pythonGH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
@StanFromIreland

Copy link
Copy Markdown
Member

CC @encukou

This needs adaptations for 3.13, which does not support Zstandard:


Traceback (most recent call last):
  File "/home/runner/work/cpython/installed-python/lib/python3.13/unittest/loader.py", line 403, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/runner/work/cpython/installed-python/lib/python3.13/unittest/loader.py", line 346, in _get_module_from_name
    __import__(name)
    ~~~~~~~~~~^^^^^^
  File "/home/runner/work/cpython/installed-python/lib/python3.13/test/test_zipfile/test_core.py", line 2658, in <module>
    @requires_zstd()
     ^^^^^^^^^^^^^
NameError: name 'requires_zstd' is not defined. Did you mean: 'requires_zlib'?

@encukou encukou changed the title [3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362) [3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA (GH-156003) (GH-156362) Sep 3, 2026
@encukou

encukou commented Sep 4, 2026

Copy link
Copy Markdown
Member

Indeed. Removed in all lower backports, thanks!

epatey added a commit to UKGovernmentBEIS/inspect_ai that referenced this pull request Sep 9, 2026
… carrying the gh-156002 zipfile change (#5209)

* Fix zstd .eval reads on Pythons carrying the gh-156002 zipfile change

CPython gh-156002 (CVE-2026-15310, "Memory exhaustion via crafted zip file")
makes ZipExtFile._read1 call decompress(data, max_length) on non-deflate
decompressors and consult needs_input before reading more compressed bytes.
The multi-frame zstd decompressobj registered for ZIP_ZSTANDARD exposed
neither, so every zstd member read raised

    AttributeError: '_MultiFrameZstdDecompressObj' object has no attribute '_needs_input'

on interpreters that ship the change: Docker Hardened Images' python:3.13-dev
since 2026-09-03 (it applies the open 3.13 backport python/cpython#156738
verbatim), and every maintained CPython once #156737-#156741 merge.

Accept max_length without enforcing it (the zstandard backport has no output
bound; withholding bytes would add copies without bounding memory) and report
needs_input=True (decompress always drains its input and emits all output).
For this wrapper the new _read1 thereby reduces exactly to the old one.

Tests cover the new call shape on any interpreter (a double of the post-
gh-156002 _read1 loop), the unchanged one-arg shape, partial input across
frame boundaries, and read/read1/readline round trips of a multi-frame entry.

* Type the zstd decompressor test helper instead of suppressing attr-defined

The test file added seven reason-less type: ignore[attr-defined] comments,
which the suppressions ledger rejects. Return the wrapper class from the
helper (asserting on the private zipfile._get_decompressor result) so no
suppression is needed, and record the ledger shrink from the now-used import.

* Trim the zipfile fix's comments and fold its tests into one _read1 double

The three direct-call tests were redundant with the _read1 double and the
existing round trips, and the double's payload compressed to 519 bytes so
every parametrized read size took a single call. Use an incompressible
payload so 4096 slices the stream across the frame boundary and 65_536 does
not, and drop the n=1 row (identical to 4096 after MIN_READ_SIZE).

* Move CHANGELOG entry under Unreleased

---------

Co-authored-by: Eric Patey <eric.patey@gmail.com>
@encukou
encukou marked this pull request as draft September 10, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants