Skip to content

zlib: grow the output buffer of large async one-shot calls - #66863

Open
anonrig wants to merge 2 commits into
nodejs:mainfrom
anonrig:zlib-grow-output
Open

anonrig wants to merge 2 commits into
nodejs:mainfrom
anonrig:zlib-grow-output

Conversation

@anonrig

@anonrig anonrig commented Oct 19, 2026

Copy link
Copy Markdown
Member

This builds on #66854, so only the last commit (753dd48) is new here.

After #66854 the async convenience methods hand the whole input to zlib in one write, but they still take one threadpool round trip per chunkSize (16 KiB by default) of output. A zlib.gunzip() that returns 4 MiB takes 257 round trips.

Only the concatenated result of these calls is visible to the caller, so this doubles the output buffer whenever it fills up, up to 1 MiB. 4 MiB of output now takes 10 round trips. The buffer never grows past what's needed to tell that maxOutputLength was exceeded, so it doesn't decompress any further past the limit than before. Results that fit in the first buffer take the same path as before, and the stream classes aren't touched. The extra memory while a call runs is the unused tail of the last buffer, so under 1 MiB.

Tests:

Benchmarks

benchmark/compare.js, inflate.js method='inflate' with inputLen raised to get large outputs and n scaled down. My machine was heavily loaded (load average 70–100 from other builds), so the intervals are wide, but the effect is far outside them:

inputLen vs #66854, run 1 (30 runs) vs #66854, run 2 (40 runs) vs main, run 1 / run 2
1 MiB +332.45% *** +379.94% *** +359.03% *** / +349.12% ***
64 KiB +40.25% *** +47.88% ***

A/A (#66854 against itself, 1 MiB, 40 runs) is +2.97% ±20.84%.

The default 1 KiB configurations don't reach the new code, because their output fits in the first buffer. The --max-regression 5 gate over deflate.js, inflate.js, pipe.js and creation.js passed but came back inconclusive under that load (intervals around ±40%). Outside one-shot calls, the only change is an extra condition when a stream's output buffer fills up.

compare.js output
vs #66854, run 1
                                                          confidence   improvement   accuracy (*)    (**)   (***)
zlib/inflate.js n=2000 inputLen=1048576 method='inflate'  ***           +332.45 %   ±99.97%  ±134.26%  ±177.29%
zlib/inflate.js n=20000 inputLen=65536 method='inflate'   ***            +40.25 %   ±19.32%  ±25.78%  ±33.69%

vs main, run 1
zlib/inflate.js n=2000 inputLen=1048576 method='inflate'  ***           +359.03 %   ±100.05%  ±134.44%  ±177.65%

vs #66854, run 2
zlib/inflate.js n=2000 inputLen=1048576 method='inflate'  ***           +379.94 %   ±108.51%  ±145.17%  ±190.56%
zlib/inflate.js n=20000 inputLen=65536 method='inflate'   ***            +47.88 %   ±23.80%  ±31.60%  ±41.01%

vs main, run 2
zlib/inflate.js n=2000 inputLen=1048576 method='inflate'  ***           +349.12 %   ±64.07%  ±85.64%  ±112.25%

A/A, #66854 vs #66854
zlib/inflate.js n=2000 inputLen=1048576 method='inflate'                  +2.97 %   ±20.84%  ±27.65%  ±35.82%

Below, Chrome talks to three local servers, one each on main, #66854 and this PR, all with the same 4 MiB payload. It decodes a zlib.gzip() response and uploads a CompressionStream('gzip') body that the server decodes with zlib.gunzip(). The bytes match on all three. One zlib.gunzip() takes 258, 257 and 10 round trips. The per-call time is the median of five rounds, interleaved so the load hit each build alike.

Chrome checking main, #66854 and this PR with a 4 MiB payload: gzip responses and CompressionStream uploads decode to the same bytes on all three, one zlib.gunzip() takes 258, 257 and 10 threadpool round trips, and takes 45.70, 39.15 and 8.95 ms per call

The asynchronous convenience methods (`zlib.deflate()`,
`zlib.inflate()`, `zlib.gunzip()`, ...) wrote the input to a
Transform stream and then ended it. That took two threadpool round
trips, one for the data and one for the final flush, and ran the
result through the stream machinery.

The whole input is known up front, so unless `info` or a custom
`flush` is requested, process it with the finish flush directly, the
way the synchronous methods already do, and collect the output in the
write callback. A call whose output fits in one buffer now takes a
single threadpool round trip. The output is unchanged: the previous
two-write sequence already produced the same bytes as the synchronous
methods. `zstdCompress()` keeps the separate end-of-frame write,
because a single `ZSTD_e_end` call can produce slightly larger frames.

benchmark/compare.js, 40 runs, n=100000:

                                          confidence  improvement
    zlib/inflate.js method='inflate'      ***          +94.91 %
    zlib/deflate.js method='deflate'      ***          +66.27 %

Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Assisted-by: Grok
The asynchronous convenience methods hand the whole input to zlib in
one write, but still take one threadpool round trip per `chunkSize`
(16 KiB by default) of output. Decompressing 4 MiB took 257 round
trips.

Only the concatenated result of these calls is observable, so double
the output buffer whenever it fills up, up to 1 MiB. 4 MiB of output
now takes 10 round trips. The buffer never grows past what is needed
to tell that `maxOutputLength` was exceeded, and results that fit in
the first buffer are unchanged.

Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Assisted-by: Grok
@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies. labels Oct 19, 2026
@anonrig
anonrig requested review from H4ad, mcollina and ronag October 19, 2026 20:32
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.50%. Comparing base (2052d0d) to head (753dd48).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66863      +/-   ##
==========================================
- Coverage   90.50%   90.50%   -0.01%     
==========================================
  Files         797      797              
  Lines      278195   278282      +87     
  Branches    53453    53472      +19     
==========================================
+ Hits       251777   251854      +77     
- Misses      16795    16806      +11     
+ Partials     9623     9622       -1     
Files with missing lines Coverage Δ
lib/zlib.js 98.21% <100.00%> (+0.09%) ⬆️

... and 25 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 20, 2026

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is ok. Are the tests actually needed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. zlib Issues and PRs related to the zlib module and its compression dependencies.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants