Proposal: Deprecate implicit flowing of stream via on('data')

[chrisdickinson]

Via a conversation in this issue (towards the bottom of my linked comment.)

I propose that we deprecate (with the intent of eliminating) the implicit flowing of streams via .on('data'), opting instead to encourage users to use .resume() and .pipe(). It breaks the EventEmitter API contract to have side-effects based on adding a listener, which surprises users. While there are ways around it for normal listeners (namely EventEmitter.prototype.on.call(stream, 'data', fn)), they're not natural, and EventEmitter APIs that call .on will still trigger side-effect-ful behavior (for example, .once).

Benefits

• Streams would only have two states, flowing and paused, instead of the three they have currently (the undocumented third state, "undefined").
• Streams may be .pause()'d and .resume()'d without losing access to the third "undefined" state.

Obstacles

• Readable#pipe could no longer rely on the implicit flowing behavior of .on('data') to start the stream (so we would likely have to call src.resume()).
• streams1.Stream#pipe is entirely separate from `Readable#pipe. This may or may not prove to be a problem since streams1 streams implicitly flowed to begin with.
• Messaging/education – we don't want people to get streamsN fatigue.
• In general in io.js/node development, it's hard to tell how many packages will break due to a given change, or even to get a read on the number of users of a given feature.

Questions

• Is the simplification of the streams state machine worth the effort (for us as well as for end users)?
• In general, is making the API more explicit desirable? If so, what would it look like if .resume() was the only method to start flowing a stream? I.E., if this class of change represents "a good thing," why not take it even further?

/cc @iojs/streams

[grncdr]

Speaking as a user of streams and an author of libraries that use them, I think getting rid of the implicit flowing of data with .on('data') makes a lot of sense for the exact reasons you've described.

In general, is making the API more explicit desirable? If so, what would it look like if .resume() was the only method to start flowing a stream? I.E., if this class of change represents "a good thing," why not take it even further?

This is an interesting proposition. Am I correct in thinking that .resume() would propagate correctly through a pipeline using the same back-pressure mechanisms already in place? For example:

var pipeline = fs.createReadStream('somefile.bin')
  .pipe(someParser())
  .pipe(someOtherTransform())
  .pipe(fs.createWriteStream('somefile.bin'))

// the entire pipeline will flow here:
pipeline.resume()

If that's the intended behaviour then I think it makes a ton of sense.

[sonewman]

I guess this would promote the use of stream pipelines rather than allowing streams to have listeners. Personally i use on data a lot still (but i do use streams3) however I am not opposed to this idea, since it would allow more explicit control over the streams flowing behaviour (more like a tap you have to manual turn on, rather than one you put your hands under, which then starts pouring on you).

The only thing is that the word 'resume' suggests that it has already been started and stopped. Where as it would actually be starting the stream in the first instance. (I am not suggesting we need a start method however).

I think it would be a breaking change, and would probably require a major version bump of io.js. But I am pro to the idea of removing "magic" (in this case the monkey patching of .on) to improve user predictability.

We do some similar magic with the readable event too. What are your thoughts on that?

I guess in that instance it is different though, because otherwise there is a chance a readable event could be emitted when nothing is listening, and therefore it would be difficult to know if .read() could be called

[vkurchatkin]

+1. Patching addListener is wrong.

[isaacs]

If we remove the implicit start when a data handler is added, that'll break almost every stream library and program written to date.

Do some research on the impact of this. You'll find that many many programs and libraries in the wild are built on the assumption that data starts flowing without calling resume.

It is completely wrong to break users' programs in the name of ideological purity. It would be a disaster.

[sonewman]

@isaacs you make a perfectly valid point, and it was certainly my initial concern.

I use on data almost always (except in an exclusive read situation).

Although I am not adverse to idea bouncing and experimentation, we need clear collaboration and discussion about the best direction / roadmap for streams, as I think we could be getting too fantastical in terms of where to take this.

@chrisdickinson stated this a proposal and whether devils advocation was intentional or not this is exactly the kind of conversations we need to be having as a group of contributors community 😃.

[chrisdickinson]

@isaacs it is ideological, in that it's catering to the principle of least surprise. The goal is to narrow down the number of places one has to say "oh, but remember that..." when working with streams.

Re: research: yes, I'll do some and post the results here. Are there any packages you would particularly recommend testing? I'm thinking of just scraping the dependents of through and through2 + the major frameworks.

I may have also misrepresented the issue a bit by putting the action item in the title but describing the end result in the description: I am not suggesting we immediately drop .on('data') for flowing, but that we consider heading in that direction, by adding a deprecation notice to streams that start flowing through that mechanism. The plan would be to deprecate first and then give ample time for the ecosystem to "heal" around the new API before removing the old mechanism.

[isaacs]

I'm just not convinced that this surgery is even necessary. Pragmatism should be our ideology. If on('data') starting the flow isn't causing any real problems for a majority of our users, I'm not convinced that removing the auto-flowing would be a net reduction in surprise to our users. Most people expect streams to flow when they start interacting with them, and not before.

If they meditate on the nature of pure event emission, then they'll see that this is presents a contradiction, but the history of streams1 shows that 0.00% of Node users ever perform that meditation. (And in fact, that is a pretty conservative number of significant digits.) I share your esthetic reaction. But the esthetic objection of a vanishingly small minority of users (even if those users are contributors and TC members) should not even factor into the equation.

If we are going to perform surgery on the stream concept, we need to first identify the eventual goal. I don't think that looks like an EventEmitter at all. We should explore stuff like FRP, promise-streams, dom-streams, etc., and figure out the ideal endgame API that will unite all the JavaScript streaming libs together. I know that @domenic has done some work getting the WhatWG to come to some agreement on streaming lib ideas.

From there, we can plot a course that is minimally hazardous for our users. My suspicion is that the ideal approach will be to implement the new idealized stream API, and then either have it exist in parallel (somehow?) or with a mode-switch of some sort (ouch) or whatever other least-bad thing we can come up with to get there.

[sonewman]

@isaacs I like the idea of meditating on streams "be like water"...

anyway, I think everything you say is right. We need a plan and collaborative vision, the closes thing so far IMO was @mikeal's comment in issue #97

Any differences between io.js streams and WHATWG streams will have to be reconciled at some point. By that I do not mean that io.js streams will become WHATWG streams or vice versa, but that we need to iron out any irreconcilable differences while WHATWG streams are being standardized so that either io.js streams become a compatibility API between legacy streams and WHATWG stream or transpilers like browserify can solve this in the future.

The thing I'm most worried about is a situation similar to what we have with Buffer vs TypedArray.

We should absolutely explore all possibilities, since the solution is likely to be the best bits of everything.

Would could also do with (a brief) yet concise technical specification of a stream (this could be derived from WHATWG). Just to be clear on our objective. I mean right now it's possible to do interesting things with the streams we have, which is not always obvious to the new or experienced user.

Lots of people are still using node v0.10.x (due to corporation confidence constraints - I know this well) and may not be using the readable-streams module v1.1.13. Therefore not even reaping the benefits of streams3.

There are obviously lots of great resources out there for using streams, but are usually based on streams2 since it has been the de facto.

@chrisdickinson I still think you should definitely continue your explorations and please keep posting ideas.

[chrisdickinson]

This looks like a pretty solid "no thanks" :)