Stream performance improvement ideas

[benjamingr]

cc @anonrig from our chat

• Reduce memory overhead by using bit maps instead of many booleans for state
• Use a deque instead of a linked list for the BufferList
• Optimize for a single pipe rather than an array of pipes and fallback to array of pipes

More wild ideas:

• Split Readable into several classes and "bailout" from fast to slow ones (e.g. objectMode and bytes, multiple readers vs. one, flowing vs. non flowing) based on most streams not changing their type during their lifetime.
• Don't require a "full" event emitter necessarily for a stream unless the user is listening to an event and "fast-path" if a stream is piped

cc @ronag @nodejs/streams

[ronag]

cc @anonrig from our chat

• Reduce memory overhead by using bit maps instead of many booleans for state

Not sure whether this is worth the effort. Is memory usage a problem? Number of streams in typical application * state size should be quite low, relatively speaking? I tried this once but the performance impact was negligible. I was thinking that being able to keep more of the state in a single cache line should make a impact but I was wrong. Might be different now...

• Use a deque instead of a linked list for the BufferList

I think adding unshift to FixedQueue would be one way to go?

• Optimize for a single pipe rather than an array of pipes and fallback to array of pipes

Not sure what this is? We already optimized for single pipe? What am I missing

More wild ideas:

• Split Readable into several classes and "bailout" from fast to slow ones (e.g. objectMode and bytes, multiple readers vs. one, flowing vs. non flowing) based on most streams not changing their type during their lifetime.

Difficult.

• Don't require a "full" event emitter necessarily for a stream unless the user is listening to an event and "fast-path" if a stream is piped

Skeptical if this is worth it.

Other stuff:

• add support in pipeline for sync transformer function
• Rewrite transform stream as a Readable which implements the Writable trait. Avoiding the double buffering from Duplex.
• Try to avoid nextTick calls. I have a PR for this.

[ronag]

Do we have any actual profile data to back our efforts?

[ronag]

TBH. I think the biggest impact we can have is to try and increase the highwatermark across the board, e.g. increasing from 16k to 128k has a theoretical perf improvement of 8x in the context of streams overhead.

[benjamingr]

Not sure whether this is worth the effort. Is memory usage a problem? Number of streams in typical application * state size should be quite low, relatively speaking? I tried this once but the performance impact was negligible. I was thinking that being able to keep more of the state in a single cache line should make a impact but I was wrong. Might be different now...

i'll open a PR and we can check but since streams are so omnipresent I would suspect so especially when people construct smaller streams (e.g. to use the iterator helpers). I feel like doing the work and seeing would be easier and I feel like I got sucked into a lot more talking than coding roles recently so that would be fun.

I think adding unshift to FixedQueue would be one way to go?

That would work

Not sure what this is? We already optimized for single pipe? What am I missing

We always store a .pipes array even though most streams only pipe to one place, we could make the default behavior (one pipe) faster and bail out on multiple pipes.

Do we have any actual profile data to back our efforts?

No, I went through the systems analyzer with --log-all and noticed some stuff (optimzing streams wasn't the meeting topic V8 was)

[ronag]

We always store a .pipes array even though most streams only pipe to one place, we could make the default behavior (one pipe) faster and bail out on multiple pipes.

I don't see how this affects any hot path.

[benjamingr]

Basically all streams become slightly smaller and lookup for the pipe becomes slightly faster which would have very small impact in "many big streams" but measurable in "many small streams"

[benjamingr]

Said PR nodejs/node#49745

[mcollina]

The main performance bottleneck in streams comes from the cost of buffering at each step of a stream pipeline. From both an algorithmic and memory perspective, this is heavy. Worsely, we do this double buffering in http, because both the socket and IncomingRequests are Readable.

We should internalize a library such as syncthrough or minipass (or one built with the same principles) that does not perform any buffering. Look at the benchmarks.

[benjamingr]

@mcollina we can explore/add a BYOB fast mode where different steps of the transform mutate the data in place instead of copying it which should ideally mean 0 copies

[mcollina]

we can explore/add a BYOB fast mode where different steps of the transform mutate the data in place instead of copying it which should ideally mean 0 copies

That is impossible with the current API. We did a full design for this a long time ago, but there was no budget to make it happen. Using synchthrough/minipass where you'd use a Transform is a good intermediate step (you'd need this anyway for a BYOB solution).

[benjamingr]

@mcollina why? We keep the same design and just emit the buffers that were passed to us, we keep a pointer to what we're currently using etc?

[benjamingr]

@ronag trying FixedQueue suggestion in nodejs/node#49754

[mcollina]

For a BYOB strategy to work, you need to "lock" the two streams sides so that others could not interject and inspect that. Our current stream implementation is designed as a one-to-many abstraction.

[benjamingr]

For a BYOB strategy to work, you need to "lock" the two streams sides so that others could not interject and inspect that.

Correct, we could for example have a base class that is not an event emitter and only pipelines (and composes etc) and listening to events on that would throw or some such. Would need a lot of thought but can happen I think?

[mcollina]

The design for that was already done in https://github.com/Fishrock123/bob. I think this was 2017-2018 work.