Https imports integrity checksums

[mrozbarry]

It would be great if node could automatically handle generating integrity checksums for new https resources if they don't exist already. I'm going to assume that most js scripts in the wild don't have their own checksums.

Also, is there a good standard in place for servers that don't want someone just importing their scripts? I'm assuming CORS won't really cover that, considering know doesn't necessarily have to have a concept of it's own server.

Originally posted by @mrozbarry in #36328 (comment)

[pimterry]

I also have some concerns/questions about HTTPS imports. I'm interested in security, but also in guarantees that my builds today will keep working tomorrow.

My concern is that if this flag is eventually enabled by default:

• It would be easy to add a dependency that internally uses a remote URL, to end up with remote URL dependencies without realising.
• When you do so, everything would work fine.
• At any future time the remote content could change (checksums help here) or disappear (game over).

Even if it's not enabled by default and you don't want to enable it - if any popular packages start using this, then you're going to have to enable it or give up on npm entirely, and it will be very difficult to manage these invisible remote dependencies.

With node & npm today, I can be fairly confident that an npm ci with a lockfile will produce code that keeps working the same way basically forever, as it's locked by both the checksum and npm's own policies against package removal or changes. Enabling remote imports breaks that guarantee.

I think there's one solution that would be interesting to provide security & reliability, going beyond checksums: node could vendor remote dependencies automatically.

I.e. when you first import a remote dependency, store the content locally in <project>/.node-imports/<hash> (for example) and keep a map in that directory from URL to hash. That way:

• Content can't change or disappear.
• It's hard to accidentally start depending on remote content via dependency without realising (changes will appear in git).
• Future imports work even if you have no internet connectivity.
• Remote servers only get one request per project install, not one per usage, preserving privacy and saving them bandwidth.

This only works if the cache is not node_modules, because that's generally git ignored.

[coder543]

As I understand it, in the Rust world, you cannot cargo publish a crate (package) that depends on any crates that aren't on the crates.io registry. I think this would be a good model for npm to follow here. If you are building an application, you could still depend on things via npm and/or https imports, but libraries would have to choose whether they are going to be an https import or an npm package published to the npm registry. This ensures that packages on the npm registry would retain their current guarantees of availability, since there would be no dependency on imports outside of the registry.

I also completely agree such http imports should have the ability (or requirement) to specify an integrity checksum. Someone on HN proposed this syntax, which seems fine.

Without an integrity checksum, Node would have to either download the dependency on startup every time, or it would have to have a confusing caching policy that would lead to unexpectedly failing to download a new version when the developer needed it to, and that's beyond the obvious security implications.

[GeoffreyBooth]

cc @nodejs/modules @nodejs/security

Many of these issues were discussed in #36328 and are either supported via policies (please read https://nodejs.org/dist/latest/docs/api/policy.html) or noted as follow-up efforts. Having a single catchall issue for all HTTPS imports security concerns isn’t very useful, so I’m going to focus this one on integrity checks (per the original post) and please open new issues for other concerns.

[GeoffreyBooth]

Also, is there a good standard in place for servers that don’t want someone just importing their scripts? I’m assuming CORS won’t really cover that, considering know doesn’t necessarily have to have a concept of it’s own server.

At the moment I don’t think this is something we plan to address. If Deno provides support for such a feature I think we would consider it, probably copying however they implement it.

[mrozbarry]

I wonder if there could be some sort of http header, like Imported-By that could be values node:<package name>, node:* or even things that include deno. I guess I sort of have flashbacks when github had some serious issues because users were directly referencing scripts hosted by github in their websites. Obviously, node would do caching, so the issue wouldn't be at the same scale or severity, but I could foresee some web admins not wanting their copy of a script being used. Anyway, having some sort of package-related http header feels like a cheap solution.

The expensive version is some sort of key signed value in the headers.

[coder543]

I don’t understand what problem you’re proposing a header would solve. What would the presence or absence of that header do? Non-standard headers rarely solve more problems than they create, IMO.

[mrozbarry]

The header would allow the server to filter who can import a filter script. Maybe the author of a script intentions that script to be private. I guess if https imports are allowed, it should have the same sort of restrictions a package has. I'm not saying the header is the right thing, I'm just spit-balling an idea of how a server could prevent it's scripts from being imported.

[coder543]

If the server responds with the content of the script at all, the header is just a courtesy that can and would be ignored by someone who wants to import it anyways. The server really should require authentication if it wants to effectively limit who can retrieve some content, as with any other server.

[mrozbarry]

Ah, an Authorization header would be better than my spit-balling. I think the resource in package.json could have some sort of way to specify http headers in general for making the request, that would cover the standard authorization plus any other way a human would want to download the file.