Circular dependencies (prevents proper `rollup`)

[danielgindi]

Rollup spits those out:

Circular dependency: node_modules\readable-stream\lib\_stream_duplex.js -> node_modules\readable-stream\lib\_stream_readable.js -> node_modules\readable-stream\lib\_stream_duplex.js
Circular dependency: node_modules\readable-stream\lib\_stream_duplex.js -> node_modules\readable-stream\lib\_stream_readable.js ->  commonjs-proxy:C:\Users\Daniel.SILVERBOLT\Desktop\git\endpoint-analyzer\node_modules\readable-stream\lib\_stream_duplex.js -> nod
e_modules\readable-stream\lib\_stream_duplex.js
Circular dependency: node_modules\readable-stream\lib\_stream_writable.js -> node_modules\readable-stream\lib\_stream_duplex.js -> node_modules\readable-stream\lib\_stream_writable.js
Circular dependency: node_modules\readable-stream\lib\_stream_writable.js -> node_modules\readable-stream\lib\_stream_duplex.js ->  commonjs-proxy:C:\Users\Daniel.SILVERBOLT\Desktop\git\endpoint-analyzer\node_modules\readable-stream\lib\_stream_writable.js -> n
ode_modules\readable-stream\lib\_stream_writable.js

It is a huge problem, and will make code unexecutable after "rolling up".
The rollup process will convert commonjs deps into "static imports", which do not support circular dependencies.
Then it will end up with a flat hierarchy where there's a reference to a (yet) undefined variable.

[mcollina]

Do you think this is a rollup problem or a readable-stream one? IMHO, it’s the former, as circular deps are valid cjs.

I’m happy to have a look anyway. Can you please produce a project/full example that demonstrate the issue?

[danielgindi]

It's kind of both. The problem is not really rollup, as what rollup does is "convert" (through a plugin) into plain ES static imports, which do not support circular dependencies.

Circular dependencies are valid in cjs - but are still problematic even there because you have to be really careful about what is really export/imported and when does the circular dependency break it.
It's really a design flaw- that could be easily fixed by lifting common code/data into separate js files.

I'll try to make up a sample project

[danielgindi]

Here is an example:

rollup-example.zip

You could run npm run build to see the build process, or just inspect the pre-compiled output at compiled.js.
Try to run it...
You'll get TypeError [ERR_INVALID_ARG_TYPE]: The "superCtor" argument must be of type Function. Received type undefined

Now look at line 1135 - you'll see the compiled util$1.inherits(Duplex$1, _stream_readable); where _stream_readable is declared only at line 1497 var _stream_readable = Readable;.
Hence _stream_readable is undefined and you get TypeError [ERR_INVALID_ARG_TYPE]:

As rollup is becoming very popular and big projects are moving there (in my opinion partly because browsers started supporting ES module system and there's much less need for transpilers), it becomes crucial to adjust modules to dump the circular dependencies, even if not adopting ES imports yet.

[targos]

Just a note: circular dependencies are valid in ESM code. It's just that they behave differently compared to CJS because files are not executed in the same order.

[targos]

This does not look like an easy case to fix:

• _stream_duplex exports the Duplex class which uses Writable.
• _stream_writable exports the Writable class which uses Duplex.

There is no common dependency that could be factored out.

[danielgindi]

Well Writable only imports Duplex in order to test if instanceof, I'm sure we can find another way to check if it's a Duplex :-)

[FabianSellmann]

Using webpacks circular dependency plugin also brings this up.

[danielgindi]

Instead of this:

function ReadableState(options, stream, isDuplex) {
  Duplex = Duplex || require('./_stream_duplex');
  options = options || {}; // Duplex streams are both readable and writable, but share
  // the same options object.
  // However, some cases require setting options to different
  // values for the readable and the writable sides of the duplex stream.
  // These options can be provided separately as readableXXX and writableXXX.

  if (typeof isDuplex !== 'boolean') isDuplex = stream instanceof Duplex; // object stream flag. Used to make read(n) ignore n and to

You could do this:

function ReadableState(options, stream, isDuplex) {
  options = options || {}; // Duplex streams are both readable and writable, but share
  // the same options object.
  // However, some cases require setting options to different
  // values for the readable and the writable sides of the duplex stream.
  // These options can be provided separately as readableXXX and writableXXX.

  if (typeof isDuplex !== 'boolean') {
    isDuplex = stream instanceof Readable && typeof stream._write === 'function' && typeof stream._writableState === 'object';
  }

With NodeJS streams, it's all about duck-typing. In order for something to be a stream it does not need to actually inherit from the base classes, it just needs to conform to a certain protocol that everyone expects.

[mcollina]

Unfortunately we cannot do that, for two reasons:

1. this modules tries to follow what Node.js core does. If we would like to change that behavior, we should seek to change it there.
2. Unfortunately these lines are not about ducktyping, they mean exactly that specific function, because it comes from these source files.

[danielgindi]

I am aware that there IS support for circular dependencies, but not the way you do them :-)
You are dynamically requiring the Duplex module, because that's easier in CommonJS. But when converting to ES6, one needs to jump through hoop to get the same theoretical behavior as in CJS.
That creates a hot mess, and neither rollup nor webpack can handle that.
You could have gone the other way around, requiring them in the top scope, but then you have to not replace module.exports, instead you have to attach the exported stuff to it. This would be more consistent with how ES6 does that, but less consistent with (current) nodejs code.

So another compromise which would not change the almost-100% nodejs compatibility, would be a global (private) registry.

Something like this:

_registry.js:
// An empty file would also do as it has an empty `exports` object, but may behave differently when converted to ES6
const registry = {};
module.exports = registry;

_stream_duplex.js:

const Registry = require('_registry');
.
.
.
Registry.Duplex = Duplex;

_stream_readable.js:

const Registry = require('_registry');


function ReadableState(options, stream, isDuplex) {
// -  Duplex = Duplex || require('./_stream_duplex');
.
.
.
  if (typeof isDuplex !== 'boolean') isDuplex = stream instanceof Registry.Duplex; // object stream flag. Used to make read(n) ignore n and to
  // make all the buffer merging and length checks go away

.
.
.
}
.
.
.

function Readable(options) {
// -  Duplex = Duplex || require('./_stream_duplex');
.
.
.
  var isDuplex = this instanceof Registry.Duplex;
.
.
.
}

Registry.Readable = Readable;

Executing the Readable constructor will only happen after all classes registered, so everyone can access everyone.