[3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA (GH-156003) (GH-156362) - #156738
Draft
miss-islington wants to merge 2 commits into
Draft
[3.13] gh-156002: Bound zipfile decompression for bzip2/LZMA (GH-156003) (GH-156362)#156738miss-islington wants to merge 2 commits into
miss-islington wants to merge 2 commits into
Conversation
…tandard (pythonGH-156003) (pythonGH-156362) Patch by @tonghuaroot. zipfile.ZipExtFile._read1() bounds the output of each decompress() call for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA, and Zstandard members it called decompress() with no bound. A whole compressed chunk was therefore expanded into a single allocation before the data[:self._left] clip ran, so a consumer that deliberately reads in small chunks to limit memory (for example zf.open(name).read(8192)) was silently unprotected for non-DEFLATE members. A small, spec-conformant archive member declaring a large uncompressed size could drive multi-GB peak memory. _read1() now passes a per-call bound to the non-DEFLATE decompress() (mirroring the DEFLATE branch) and drains the decompressor's internal buffer across calls by checking needs_input before reading more compressed input. zipfile's LZMADecompressor wrapper forwards max_length and exposes needs_input so the bound also holds for LZMA members. (cherry picked from commit f897dbf) (cherry picked from commit 1b424c0) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
This was referenced Aug 31, 2026
Member
|
CC @encukou This needs adaptations for 3.13, which does not support Zstandard: |
Member
|
Indeed. Removed in all lower backports, thanks! |
epatey
added a commit
to UKGovernmentBEIS/inspect_ai
that referenced
this pull request
Sep 9, 2026
… carrying the gh-156002 zipfile change (#5209) * Fix zstd .eval reads on Pythons carrying the gh-156002 zipfile change CPython gh-156002 (CVE-2026-15310, "Memory exhaustion via crafted zip file") makes ZipExtFile._read1 call decompress(data, max_length) on non-deflate decompressors and consult needs_input before reading more compressed bytes. The multi-frame zstd decompressobj registered for ZIP_ZSTANDARD exposed neither, so every zstd member read raised AttributeError: '_MultiFrameZstdDecompressObj' object has no attribute '_needs_input' on interpreters that ship the change: Docker Hardened Images' python:3.13-dev since 2026-09-03 (it applies the open 3.13 backport python/cpython#156738 verbatim), and every maintained CPython once #156737-#156741 merge. Accept max_length without enforcing it (the zstandard backport has no output bound; withholding bytes would add copies without bounding memory) and report needs_input=True (decompress always drains its input and emits all output). For this wrapper the new _read1 thereby reduces exactly to the old one. Tests cover the new call shape on any interpreter (a double of the post- gh-156002 _read1 loop), the unchanged one-arg shape, partial input across frame boundaries, and read/read1/readline round trips of a multi-frame entry. * Type the zstd decompressor test helper instead of suppressing attr-defined The test file added seven reason-less type: ignore[attr-defined] comments, which the suppressions ledger rejects. Return the wrapper class from the helper (asserting on the private zipfile._get_decompressor result) so no suppression is needed, and record the ledger shrink from the now-used import. * Trim the zipfile fix's comments and fold its tests into one _read1 double The three direct-call tests were redundant with the _read1 double and the existing round trips, and the double's payload compressed to 519 bytes so every parametrized read size took a single call. Use an incompressible payload so 4096 slices the stream across the frame boundary and 65_536 does not, and drop the n=1 row (identical to 4096 after MIN_READ_SIZE). * Move CHANGELOG entry under Unreleased --------- Co-authored-by: Eric Patey <eric.patey@gmail.com>
encukou
marked this pull request as draft
September 10, 2026 13:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Patch by @tonghuaroot.
zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2 and LZMA,
members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.
_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.
(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)
Co-authored-by: Petr Viktorin encukou@gmail.com
Co-authored-by: tonghuaroot tonghuaroot@gmail.com