Decode compressed response bodies incrementally - #1126
Conversation
Previously each raw chunk was fully inflated in a single `decompress()` call before being re-chunked, so a small compressed chunk could inflate to an arbitrarily large buffer and `iter_bytes(chunk_size)` did not actually bound memory. Rework the content decoders to yield bounded pieces as they decode: `gzip`/`deflate` drain a shared `ZlibDecompressor` with `max_length`, `brotli` uses `output_buffer_limit` (now requires `brotli>=1.2.0`), and `zstd` uses `max_length` on the stdlib `compression.zstd` backend. `MultiDecoder` pipes children lazily so the bound holds across stacked encodings. `iter_bytes(chunk_size)` now bounds peak memory like urllib3's `read(amt)`. Also close the underlying stream when decoding raises part-way through, so a decode error releases the connection instead of leaking it.
|
Docs preview: https://c4239a3d-httpx2-docs.pydantic.workers.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 472041bacf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Merging this PR will not alter performance
Comparing 
Footnotes
|
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 8 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…ecoding # Conflicts: # src/httpx2/httpx2/_decoders.py
1 participant
Summary
The content decoders used to inflate each raw chunk fully in a single
decompress()call and re-chunk the result afterwards, so a small compressed chunk could inflate to an arbitrarily large buffer anditer_bytes(chunk_size)did not actually bound peak memory.This reworks the decoders to yield bounded pieces as they decode:
gzipanddeflatedrain a shared zlib decompressor withmax_length.output_buffer_limit, adjusted for the C backend's allocation behavior.max_lengthfromcompression.zstdorbackports.zstd.MultiDecoderpipes its children lazily, so the bound holds across stacked encodings.Each decode step is bounded to 1 MiB (
MAX_DECODE_CHUNK_SIZE).iter_bytes(chunk_size)andaiter_bytes(chunk_size)now bound peak memory rather than materializing a whole decoded chunk before re-slicing it.The change also closes the underlying stream when decoding raises part-way through, so a decode error releases the connection instead of leaking it.
Validation
scripts/checkscripts/test && scripts/coverage- 1,990 passed, 1 skipped, 100% coverageAI Disclaimer
This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.