Skip to content

zlib: decode concatenated zstd frames - #65911

Closed
agape1225 wants to merge 1 commit into
nodejs:mainfrom
agape1225:zlib-zstd-concatenated-frames
Closed

zlib: decode concatenated zstd frames#65911
agape1225 wants to merge 1 commit into
nodejs:mainfrom
agape1225:zlib-zstd-concatenated-frames

Conversation

@agape1225

Copy link
Copy Markdown
Contributor

A single write to a zstd decompression stream can hold several
concatenated zstd frames. The zstd format explicitly allows this and
zstdcat decodes it, but Node stopped after the first frame whenever
the frame boundary fell inside one input buffer:

const zlib = require('node:zlib');
const data = Buffer.concat([
  zlib.zstdCompressSync(Buffer.alloc(2000, 0x41)),
  zlib.zstdCompressSync(Buffer.alloc(2000, 0x42)),
]);
zlib.zstdDecompressSync(data).length; // 2000, expected 4000

ZstdDecompressContext::DoThreadPoolWork ran a single
ZSTD_decompressStream call, and the JS processing loop treated the
leftover input with free output space as the end of the stream. gzip
already handles concatenated members correctly, and the reporter of
the issue hit this in practice after ~3 GB of streamed data.

This keeps decoding across frame boundaries while input is pending and
output space is left, mirroring the gzip multi-member loop. When
rejectGarbageAfterEnd is set, it stops after the first frame so the
trailing frames are still reported as junk — the option is now wired
through to the zstd decompression context for that purpose.

rejectGarbageAfterEnd: false + trailing bytes that are not a valid
frame now produces a decode error instead of being silently ignored,
which matches gzip's behavior for trailing non-padding garbage.

Fixes: #64741

This re-does #64748 by @lazerg (closed for lack of review) on top of
current main, keeping the same approach; credited via Co-authored-by.

🤖 Generated with Claude Code

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies. labels Sep 8, 2026
A single write can hold several concatenated zstd frames, which the
zstd format explicitly allows and which `zstdcat` decodes. When the
frame boundary fell inside one input buffer, the zstd decompress
context ran a single decode call, and the processing loop treated the
leftover input with free output space as the end of the stream, so the
remaining frames were dropped.

Keep decoding across frame boundaries while input is pending and
output space is left, mirroring how the gzip path already handles
concatenated members. When `rejectGarbageAfterEnd` is set, stop after
the first frame so the trailing frames are reported as junk; the
option is now wired through to the zstd decompression context for
that purpose.

Fixes: nodejs#64741
Refs: nodejs#64748
Co-authored-by: Lazizbek Ergashev <lazerg2@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: agape1225 <49804691+agape1225@users.noreply.github.com>
@agape1225
agape1225 force-pushed the zlib-zstd-concatenated-frames branch from 0fb78f4 to f93a2bb Compare September 8, 2026 12:50
@jasnell

jasnell commented Sep 8, 2026

Copy link
Copy Markdown
Member

duplicates #65865

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.18%. Comparing base (2987a59) to head (f93a2bb).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
src/node_zlib.cc 90.00% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #65911      +/-   ##
==========================================
+ Coverage   90.16%   90.18%   +0.02%     
==========================================
  Files         771      771              
  Lines      265097   265077      -20     
  Branches    50358    50344      -14     
==========================================
+ Hits       239026   239062      +36     
+ Misses      17011    16963      -48     
+ Partials     9060     9052       -8     
Files with missing lines Coverage Δ
lib/zlib.js 98.14% <100.00%> (+<0.01%) ⬆️
src/node_zlib.cc 79.72% <90.00%> (+0.19%) ⬆️

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@agape1225 agape1225 closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zstd decoding complete frame in read stream halts stream

3 participants