fix: glob subdirectories in server/chunks to support turbopack builds

[timneutkens]

Description

TLDR: This change makes next build --turbopack work on Netlify

Was investigating why next build --turbopack fails when deploying on Netlify:

• Turns out this glob is targeting .next/server/chunks/* instead of .next/server/chunks/**/* in the standalone dir.
• With Turbopack the output can have additional subdirectories (could have happened with webpack too if the output folder was ever changed).
• Upon further investigation it seems that you'd want to collect all of .next/standalone/.next not just .next/* and .next/server/**/*
• I don't think there should be a limit on .js because that would exclude e.g. sourcemaps which are .js.map files.

For now to verify if there's a specific reason the subdirectories were targeted I've changed the glob to .next/server/**/* in order to see if any tests fail.

Documentation

Tests

TODO: This needs a next build --turbopack app.

You can test this change yourself like so:

1. TODO

Relevant links (GitHub issues, etc.) or a picture of cute animal

[timneutkens]

I've added a test fixture and copied the simple test suite. However I'm unable to run it successfully locally, it seems I may be missing something, let me know if you're able to run it 🙏

[pieh]

Firstly, huge thanks for PR!

• Turns out this glob is targeting .next/server/chunks/* instead of .next/server/chunks/**/* in the standalone dir.
• With Turbopack the output can have additional subdirectories (could have happened with webpack too if the output folder was ever changed).

Awesome find, thank you!

• Upon further investigation it seems that you'd want to collect all of .next/standalone/.next not just .next/* and .next/server/**/*
• I don't think there should be a limit on .js because that would exclude e.g. sourcemaps which are .js.map files.

Limiting to .js in .next/server/app|pages was done to not bundle .html, .rsc, .meta, .json for content prerendered at build, because we do have custom CacheHandler that doesn't use filesystem for storage so those files would needlesly grow size of produced function without being actually used. Limiting to .js like we do now is probably too restrictive overall and instead probably we should exclude those specific targeted things we do have alternative handling for.

Expanding a bit on selective dirs globbing - it was done to try to limit chances of exceeding function max size. We do generate single function for all routes today (hopefully to be changed with adapters), so it's easier in this setup to go over the edge than with function per route. This makes this change a bit risky for us and instead I'd like to suggest following:

Could we keep this change to only target turbopack builds needs for right now? My understanding just from reading first 2 points is that it could be addressed by adjusting our server/chunks/* (and server/edge-chunks/*?) pattern to allow sub directories.

Then I would follow up with some ~telemetry addition to asses impact of doing the change in this PR almost as-is (just with added exclussion for default filesystem CacheHandler entries) on size of produced lambda to understand if this is generally safe change for us to roll out or would this result in failures (function size too large) for some users.

I'm happy to take over getting this over the finish line if you wish, just let me know what your thoughts are on this.

[pieh]

Limiting to .js in .next/server/app|pages was done to not bundle .html, .rsc, .meta, .json for content prerendered at build, because we do have custom CacheHandler that doesn't use filesystem for storage so those files would needlesly grow size of produced function without being actually used. Limiting to .js like we do now is probably too restrictive overall and instead probably we should exclude those specific targeted things we do have alternative handling for.

I was checking out failing test and seems to me that those failures are actually caused by including those CacheHandler entries in function without making adjustments elsewhere. To be fair the handling is a bit brittle and not resiliant to changes like that, but it was operating under assumption that .html files would not be bundled and our fs/promises.readFile would throw -

opennextjs-netlify/src/run/next.cts

Lines 96 to 126 in d98f5b2

	try {
	// Attempt to read from the disk
	// important to use the `import * as fs from 'fs'` here to not end up in a endless loop
	return await ofs.readFile(path, options)
	} catch (error) {
	// only try to get .html files from the blob store
	if (typeof path === 'string' && path.endsWith('.html')) {
	const cacheStore = getMemoizedKeyValueStoreBackedByRegionalBlobStore()
	const relPath = relative(resolve('.next/server/pages'), path)
	const file = await cacheStore.get<HtmlBlob>(relPath, 'staticHtml.get')
	if (file !== null) {
	if (file.isFullyStaticPage) {
	const requestContext = getRequestContext()
	// On server initialization Next.js attempt to preload all pages
	// which might result in reading .html files from the file system
	// for fully static pages. We don't want to capture those cases.
	// Note that Next.js does NOT cache read html files so on actual requests
	// that those will be served, it will read those AGAIN and then we do
	// want to capture fact of reading them.
	const { initializingServer } = initAsyncLocalStorage.getStore() ?? {}
	if (!initializingServer && requestContext) {
	requestContext.usedFsReadForNonFallback = true
	}
	}
	return file.html
	}
	}
	throw error
	}

I'll attempt to open PR with optimistic change that scales back the globbing part to what seems like turbopack builds specific needs to verify that is indeed what's needed

[timneutkens]

Only matching chunks/**/* would work yeah 👍

[pieh]

Thank You!