002 - web compat of file extension/index searching

[bmeck]

In talks with @domenic a concern about using error behavior of a URL derived from step 2 of https://html.spec.whatwg.org/multipage/webappapis.html#resolve-a-module-specifier . In particular, the current plan has been to search for file extension such that:

import './foo';

Would search multiple file extensions .mjs, .js, .json, and .node.

Since ./, ../, and / do not immediately error upon resolution (they would error upon failed fetch) the question is if path searching could be limited to the error behavior for so called "bare" URLs within step 2.

[domenic]

In particular, I'm concerned about the gratuitous incompatibility that the file-extension and index.js searching introduces with browsers.

Stated another way: browsers do not have a solution for prefix-less imports yet (e.g. import "foo"). But, we will almost certainly add something in the future. The hope of doing so is that then browsers and Node.js can have 100% compatible semantics. But with the current proposal, that is impossible, because of the departure from browsers in prefix-ful imports (e.g. import "./foo").

If a Node.js author writes a file

import "foo";
import "./foo";

then the first line is not browser-compatible for now, but the second line will never be browser-compatible, because browsers will always interpret that as a literal request for ./foo, and will never issue 8 HTTP requests (4 for ./foo.extension, and 4 for ./foo/index.extension).

If the proposal instead restricted the path search magic to only prefix-less imports, then it will be much easier to guarantee compatibility across the ecosystems. Indeed, it means that any Node.js package which does not have external dependencies would work right out of the box in browsers on day 1.

Avoiding path-searching also has the minor benefit for Node.js authors of settling on a single convention (some of my projects are a mix of require("./foo") and require("./foo.js"), which is annoying). And it has the major benefit of avoiding several rounds of file I/O during program startup.

[bmeck]

My concern is that while it does settle on one convention it limits refactoring within a single project, but this is contained fairly well since very few (if any?) packages don't control the directory structure or names of their own files. I think it is acceptable as the main benefits of searching come from inter-package dependencies which still are preserved.

This does have a slight annoying affect of vendored modules outside of node_modules would no longer work the same, but people are already encouraged to use bundledDependencies for this purpose.

[ljharb]

Being able to avoid specifying the extension is critical - both in browsers and node - to ensuring easier refactoring. It's a common practice, for example, to use server rewrites to avoid including a ".php" or similar extension in all browser-side URLs, so as to not couple the browser code to the server's choice of PHP. Similarly, in node, omitting the extension means that I'm not coupling my import to my choice of .js, .mjs, .jsx, etc, which allows much more free refactoring.

The same is true with a browser requesting the URL /foo vs /foo/index.html - /foo allows for the common practice of a server rewrite to ensure that this hits foo.html or foo/index.html, and the server searches to find these.

In other words, "browser-compatible" in practice includes server configuration too, and "rewrite rules/searching" is something most servers already do.

Just to clarify, we're talking about whether the searching behavior that applies to relative paths (starting with . or /) should apply?

If we're talking about non-relative paths, these "prefixless imports" are the most common import/require path in node altogether, because they refer to npm-installed modules, so I'm not sure how you could write anything meaningful in node without the searching behavior.

If we're talking about relative paths, then we're severely limiting the ability to transparently move a relative directory in a project and extracting it out into an npm-installed module, because the way that directory is required/imported will drastically change upon moving it. This will have the effect of limiting people's ability to extract code into separate modules.

[domenic]

"rewrite rules/searching" is something most servers already do.

[citation needed]; this is not true in my experience (e.g. it is impossible on GitHub pages and other static hosting sites).

If we're talking about relative paths, then we're severely limiting the ability to transparently move a relative directory in a project and extracting it out into an npm-installed module, because the way that directory is required/imported will drastically change upon moving it.

It will literally require adding either ".mjs" or "/index.mjs". That is hardly a "drastic change" that is "critical" to ensure easier refactoring. It is certainly not worth such browser-incompatibility.

[unional]

Hosting services like apache and iis all provides rewriting rules. It is the responsibility of the server to handle this.

Mandating extension does have significant impact on development and tooling. Consider other languages that complies to JS: TypeScript, CoffeeScript, Dart, etc.

Using TypeScript for example, this could force the compiler to modify the reference from ./foo.ts to ./foo.js or ./foo.tsx to ./foo.jsx/./foo.js

Then consider running TypeScript code directly using ts-node or systemjs/jspm. The compiler need to disable such in these scenarios.

It is a mix of concern. It does seem like leaving the rewriting to the server is a better approach.

[ljharb]

Fair point, I have no evidence for "most" (although I've not used a framework that lacks it), and static hosting sites obviously lack this ability - but for static hosting sites, it's pretty common to have a build step beforehand (Github pages has this for the automatic generator, for example).

That is hardly a "drastic change"

It tightly couples consumers to the format you choose to use. Tight coupling is drastic, even if the fix it necessitates is minimal.

[domenic]

It is a mix of concern. It does seem like leaving the rewriting to the server is a better approach.

The issue is that the vast majority of people do not control their server configs. The story for creating modules that work across Node and the browser should not be: (a) install the npm package; (b) go talk to your server admin and ask them to add these n rewrite rules which the package has hopefully documented in its readme, but more likely you'll have to figure out by hand. (Also, never try to use the package on GitHub pages or other static hosting sites.)

It should just be (a).

[domenic]

but for static hosting sites, it's pretty common to have a build step beforehand (Github pages has this for the automatic generator, for example).

The whole point of ES modules in the browser, unlike CommonJS, is that there is no build step.

[domenic]

It tightly couples consumers to the format you choose to use. Tight coupling is drastic, even if the fix it necessitates is minimal.

I hope you recognize this is an opinion. We could just as easily say that tightly coupling yourself to Node's search-the-filesystem algorithm is "drastic"ally bad.

[ljharb]

"no build step" does not mesh with the way it's going to be used in practice. "minification" is a build step too, and I'd be surprised if there were many popular sites - static or otherwise - that did not minify their code.

I hope you recognize this is an opinion.

Indeed, it's an opinion - we're all stating opinions here, including that "no build step" is desirable, and that "node and the browser doing slightly different things" is problematic, and that such incompatibility is "gratuitous".

[joshgav]

Since ./, ../, and / do not immediately error upon resolution (they would error upon failed fetch)

They would error at resolution time if they can't be parsed as a relative URL to the module base path, same as absolute URLs. They would error on fetch if the script isn't actually found at the resolved path, also same as absolute URLs. Right?

If that's correct, it seems the question and decision of whether to perform additional path searches should apply the same to both absolute (algorithm step 1) and relative (algorithm step 3) URLs.

@bmeck what's our plan for non-URLs, like require('express')? The spec you referenced would make those invalid. Will Node continue to allow them? Perhaps we could treat them as though they were prepended with ./ and try base URLs from NODE_PATH, as in step 3 in the Node modules doc.

[domenic]

They would error at resolution time if they can't be parsed as a relative URL to the module base path, same as absolute URLs. They would error on fetch if the script isn't actually found at the resolved path, also same as absolute URLs. Right?

I can't think of any string that starts with those characters that will fail to parse as a relative URL.

If that's correct, it seems the question and decision of whether to perform additional path searches should apply the same to both absolute (algorithm step 1) and relative (algorithm step 3) URLs.

Agreed. And it seems bad IMO, if you do

import "file:///C:/foo";

to also search for file:///C/foo.js, file:///C/foo.mjs, file:///C/foo.json, etc. I.e. when given a URL, we should not construct a bunch of other URLs from it.

Similarly, specifying

import "./foo";

should also not construct a bunch of other URLs from it and search them.

[bmeck]

They would error at resolution time if they can't be parsed as a relative URL to the module base path, same as absolute URLs. They would error on fetch if the script isn't actually found at the resolved path, also same as absolute URLs. Right?

Correct, the behavioral change is when the error occurs. Without searching, at resolution time the fetch destination is set so initial fetch produces the error. With searching, the resolution cannot be finished until fetch has occurred.

If that's correct, it seems the question and decision of whether to perform additional path searches should apply the same to both absolute (algorithm step 1) and relative (algorithm step 3) URLs.

Unclear where these numbers are coming from.

@bmeck what's our plan for non-URLs, like require('express')? The spec you referenced would make those invalid. Will Node continue to allow them? Perhaps we could treat them as though they were prepended with ./ and try base URLs from NODE_PATH, as in step 3 in the Node modules doc.

We support those, this issue does not attempt to change behavior of "bare" URLs. We are only talking about step 2 of https://html.spec.whatwg.org/multipage/webappapis.html#resolve-a-module-specifier .

Our goal is to have an algorithms where both browsers and node can be act the same. The approach is to only differ from the browser spec in ways that would produce errors (either by occupying invalid resolution or errors during fetch).

[bmeck]

@domenic

I agree the behavior being different if a specifier parses as a URL and creating a relative/absolute from a specifier and base URL is bad. Current implementation however does not perform searching if given a fully specified URL that parses immediately. This follows the ongoing rewrite of the spec for node impl of ESM