Strict installs with node-modules linker (& the role of nmMode and nmHoistingLimits settings)

[JanJakes]

Hi, first of all, big thanks to the Yarn team for all the great work and congrats on the awesome Yarn v3 release!

This may be in part a question, in part a clarification of some functionality, and in part maybe also a feature suggestion.

Strict installs with node-modules?

We've seen different approaches to module installs and linking in the npm/yarn/pnpm ecosystems and looking at Yarn as the pioneer of strictness and pnp, I have a question:

Is currently it possible to use nodeLinker: node-modules and achieve strict installs?

By "strict installs" I mean the same module resolution behavior as pnp would provide and probably a similar one as pnpm does ("a non-flat node_modules, so code has no access to arbitrary packages"), although I know it's a bit more complicated and depends on hoisting settings, but what I mean is the strict settings.

Why?

I think having strict installs with node-modules linker may be a very good step on the road to pnp (and the perfect option for mixed pnp/node-modules setups).

Moreover, it all seems to be related – strictness, linking, hoisting, and install speeds.

For instance, if content-addressable store is used, there may be no need for hoisting and all the related complexity, and that, in turn, will provide stricter installs and better speeds (skipping hoisting computations and using hardlinks).

Will pnpm-style linker solve it?

I know the release notes mention that the pnpm-style linker is planned so my next question is – will it provide strictness with node-modules?

(If I'm not mistaken, neither nmHoistingLimits: workspaces nor nmHoistingLimits: dependencies achieve that in full, although I'm not fully sure based on what the docs of these options mention.)

What about nmHoistingLimits and hoisting in general?

It seems like nmHoistingLimits bring some more strictness but I'm not fully sure from the rather brief docs for this feature what exactly will be the effect of these.

It seems like with nmHoistingLimits: dependencies we may be getting similar strictness as we can get with pnpm without hoisting but I'm not fully sure about it in didn't try it out in depth.

My question here is what are the practical impacts of nmHoistingLimits: dependencies:

1. Can I access only listed deps from my app?
2. Can my deps access only their deps or anything higher up the tree?

I've noticed that using nmHoistingLimits: dependencies makes linking about twice as slow on a project with multiple workspaces, but I guess it may also relate to hardlinks usage and that could be my next question...

Using nmMode: hardlinks-local vs. nmMode: hardlinks-global

From the docs it seems to me that hardlinks-local uses hardlinks only for exact same packages in exact same version (that's interesting to me as there are no folder hardlinks so this still has to be done file-per file, right?) and hardlinks-global links per-file and globally on a machine, not within a project (where is that cache stored?).

I'm kind of missing a third option that sounds the most natural to me – linking per file within a project (like pointing all the hardlinks somewhere into .yarn, for instance), which is probably similar to using nmMode: hardlinks-global and pointing the global cache to local .yarn (if that's even possible).

I'm actually not sure why two different linkers need to exist here – could the local linker be just a global linker pointing to local .yarn directory? Or what are the actual differences?

But maybe I simply misunderstood the docs about how these two options work. Maybe @larixer would correct me?

(From a quick try using these linkers doesn't seem to solve the slower speeds when used with nmHoistingLimits: dependencies.)

Wrapping it up

It seems to me that there are many possible configurations using different combinations of nmMode and nmHoistingLimits but just from the docs I'm not sure if I understand those correctly.

Ultimately, when using pnp is not possible, I think the ideal setup for me would be to have fully strict node-modules linking with content-addressable store and no hoisting at all (and also no hoisting-related complexity and issues).

Will the pnpm-style linker @arcanis is working allow for such installs? Or am I misunderstanding some of the current abilities of Yarn v3 and it's already possible (or never will be)?

Thanks for any replies and clarifications and sorry for such a long post!

[larixer]

Hi, thank you for this post - many interesting points have been raised! Let's start!

Strict installs with node-modules?
By "strict installs" I mean the same module resolution behavior as pnp would provide and probably a similar one as pnpm does ("a non-flat node_modules, so code has no access to arbitrary packages"), although I know it's a bit more complicated and depends on hoisting settings, but what I mean is the strict settings.

First of all, the priority for node-modules linker is compatibility with existing npm packages, strictness comes second. It means that we cannot change the layout of node_modules in ways that will require changing more than 0.1% of existing npm packages.

The dependency graph declared by a set of package.json files of your project must be somehow represented on the disk. PnP represents the graph programmatically inside the .pnp.cjs file, this gives it maximum flexibility and allows it to deal with the loops and repetitions in the dependency graph. The node-modules linker must represent each package as a folder inside node_modules directory. To correctly represent the underlying graph on the disk and remove repetitions and loops from the graph it uses hoisting and symlinks. The symlinks are used in exceptional cases, to link workspaces to each other and the symlinks are not handled generally well by existing npm packages. The nm linker must limit the usage of symlinks as much as possible to maintain compatibility.

Hoisting crosses boundaries between packages and allows non-strict accesses during runtime. The symlinks break compatibility with existing npm packages. Can we not use symlinks and not hoist and be strict with nm linker? Well, unfortunately when we are not using hoisting the number of duplicate packages in the tree starts to grow very fast, killing installation performance and occupying disk space. We can solve the problem with the disk space growth by using hardlinks, because duplicated files are the same, but we will still have a growing performance penalty because each hardlink creation takes time.

Because of the above, strict installs with nm linker in the sense of pnpm-strictness are not possible. Note, that pnpm strictness is lower than pnp strictness (because pnpm still allows accessing undeclared dependencies from parent folders)

For instance, if the content-addressable store is used, there may be no need for hoisting and all the related complexity, and that, in turn, will provide stricter installs and better speeds (skipping hoisting computations and using hardlinks).

If you will not use hoisting, you will end up creating millions of hardlinks to content-addressable store in order for your install to be strict. Or you must use symlinks as pnpm does, but you will reduce compatibility with existing npm packages.

I know the release notes mention that the pnpm-style linker is planned so my next question is – will it provide strictness with node-modules?

It will have better strictness than nm linker, worse compatibility with existing npm packages and worse strictness than pnp. Its somewhere in the middle by all these metrics.

My question here is what are the practical impacts of nmHoistingLimits: dependencies:
Can I access only listed deps from my app?
Can my deps access only their deps or anything higher up the tree?

AFAIK, with nmHoistingLimits: dependencies you will have the same level of strictness as pnpm with default settings (version 5 and higher).
You can access only listed dependencies. Your dependencies can access anything higher up in the tree. If your workspace has node_modules in some of the parent folders it will be able to access them - the same is true for pnpm.

I've noticed that using nmHoistingLimits: dependencies makes linking about twice as slow on a project with multiple workspaces, but I guess it may also relate to hardlinks usage and that could be my next question...

It is all about trade-offs - if you want to consume any npm package in your project and want your project to be strict, use nmHoistingLimits: dependencies, yes, it has a price in higher disk space usage and installs perfs reduction. Nothing comes for free, everything is a trade-off.

From the docs it seems to me that hardlinks-local uses hardlinks only for exact same packages in exact same version (that's interesting to me as there are no folder hardlinks so this still has to be done file-per file, right?) and hardlinks-global links per-file and globally on a machine, not within a project (where is that cache stored?).

Right. Current hardlinks-local implementation is only 10% less efficient than hardlinks-global in terms of disk space consumption. Some figures would be useful to understand the differences. On one of my project I run installs with different modes and got the following:

• classic: 46secs, nm size: 2.6G
• hardlinks-local: 45 secs, nm size: 1.5G
• hardlinks-global: first install 76secs, second install 46secs, nm size: 1.3G

I'm kind of missing a third option that sounds the most natural to me – linking per file within a project (like pointing all the hardlinks somewhere into .yarn, for instance), which is probably similar to using nmMode: hardlinks-global and pointing the global cache to local .yarn (if that's even possible).

The problem with this option is that the number of files that need to be written is doubled compared to hardlinks-local and this will have a hit on performance (basically the same hit that hardlinks-global have now on a very first install). While for hardlinks-global having more files is okay, because over time only a few new files will be added to global store, reducing performance hit more and more when installing more new projects that use global store.

I'm actually not sure why two different linkers need to exist here – could the local linker be just a global linker pointing to local .yarn directory? Or what are the actual differences?

Because you might not always want to deal with global store risks when you modify a file inside node_modules of one project and changes are reflected in another project. You can remove node_modules and rerun yarn to fix this, but you need to remember at all times that the global store is used and it comes with these risks. Another annoyance is that hardlinks cannot be created across different devices, you need to keep an eye to place global cache folder and project folder on the same device. All of these are true for pnpm too

Ultimately, when using pnp is not possible, I think the ideal setup for me would be to have fully strict node-modules linking with content-addressable store and no hoisting at all (and also no hoisting-related complexity and issues).

Yes, it sounds like pnpm-style linking. On the flip side, there will be reduced compatibility with existing npm packages.

[arcanis]

Note, that pnpm strictness is lower than pnp strictness (because pnpm still allows accessing undeclared dependencies from parent folders)

Another interesting thing is that while pnpm-style installs are a bit stricter than typical nm, they aren't enough. For example, if you add foo as dependency at the root of your monorepo, all your workspaces will be able to access it. Since the pnpm strategy still relies on the filesystem, there is no way to prevent that from happening - there's no way to "hide" the root node_modules in a place where Node won't find it.

Another interesting thing is that node_modules-style installs, even the pnpm one, can't support some edge cases in peer dependency resolution, essentially ignoring intra-workspaces peer dependencies. Those cases don't happen often since you tend to keep the same dependencies in all your workspaces, but once you start to diverge the debugging can be quite annoying ...

So to answer your post: PnP is the only design that can provide perfect semantics & enforced boundaries. The node_modules design has fundamental flaws that even smart implementations like the pnpm strategy can't workaround and that make a "node_modules strict mode" impossible. In the end the pnpm benefits are:

• Doesn't require a resolver (which is less a problem nowadays, except for ESM)
• Doesn't require as much I/O as typical node_modules installs
• Prevents 90% of invalid access

On the other hand:

• Still requires a lot of I/O during installs (less than nm, but still)
• Cannot work well with zero-installs
• Cannot express full dependency tree semantics (peer dependencies)
• Cannot prevent the remaining 10% of invalid access
• Cannot throw detailed error messages (PnP can tell you exactly why a package can't be accessed)

[JanJakes]

Thanks a lot for such quick and detailed responses! So much interesting information in there!

It seems I've been mixing a few things together indeed and somehow assumed that pnpm uses hardlinks and those come at only little performance cost.

I totally missed that pnpm actually uses symlinks (and didn't know they cause compatibility issues) which explains a lot of things and everything starts to make more sense to me.

Another thing I missed is that creating hardlinks comes at a performance cost and knowing this, the difference between hardlinks-local (linking within node_modules themselves) and hardlinks-global makes sense now.

Many arguments in favor of using PnP that I wasn't aware of were mentioned in your posts. Ultimately, it seems that to have packages in a complete order and profit from the best possible performance (or even zero-installs) at the same time, PnP is the way to go and may be what I'm looking for.

---

It would be great to have a page in the docs explaining all of this (differences between linkers with regards to hardlinks, symlinks, performance, strictness, correctness of the installs, compatibility, etc., and to see how PnP can solve all of these issues). I think there are many others that don't understand all of these concepts in detail and don't know all the advantages of PnP. From the docs, I also wasn't sure what all the possible values of nmMode and nmHoistingLimits exactly do. But I know writing docs can be a lot of hard work...

Maybe all of the options can be nicely summarized as well:

	size	speed	strictness	compatibility
nmMode: classic	large	slow	low	good
nmMode: hardlinks-local	medium	medium	low	good
nmMode: hardlinks-global	medium	medium	low	good
pnpm (symlinks)	medium	fast	medium	bad
PnP	small	fast	high	?

But I'm failing to include nmHoistingLimits in there... and this way we'd get quite a big table:

	size	speed	strictness	compatibility
nmMode: classic, nmHoistingLimits: none	large	slow	low	good
nmMode: classic, nmHoistingLimits: workspaces	large	slow	low/medium	good
nmMode: classic, nmHoistingLimits: dependencies	large	slow	medium	good
...

These are indeed some things that many of us, Yarn users, may not be aware of.

---

Finally, I'm highlighting a few of the very interesting things that were mostly new to me:

Symlinks:

The symlinks are used in exceptional cases, to link workspaces to each other and the symlinks are not handled generally well by existing npm packages. The nm linker must limit the usage of symlinks as much as possible to maintain compatibility.

Hardlinks:

...but we will still have a growing performance penalty because each hardlink creation takes time.

Strictness:

Note, that pnpm strictness is lower than pnp strictness (because pnpm still allows accessing undeclared dependencies from parent folders)

Compatibility:

Or you must use symlinks as pnpm does, but you will reduce compatibility with existing npm packages.

Whoa 🙀:

Another interesting thing is that node_modules-style installs, even the pnpm one, can't support some edge cases in peer dependency resolution, essentially ignoring intra-workspaces peer dependencies.

[JanJakes]

Seeing #3338 merged, I have one more question. In your answers you cleared it up to me that pnpm-style linker reduces compatibility due to the usage of symlinks. My question is – how significant this is? Do I run into this when building a simple React app or a Node.js-powered backend or does it occur only in some edge cases? What are some examples of popular but incompatible libraries? Thanks!

[merceyz]

What are some examples of popular but incompatible libraries?

It's not really a library but to give a high profile example, React Native does not follow symlinks facebook/metro#1