Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure (#4240)

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:
1) A persistently running frontend server is initialized once when a build starts.
2) build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
3) When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
4) The frontend server processes compilation requests and serves compiled JS files back to `build_runner` (hot-reload ready).

Major changes:
* Adds a `DdcFrontendServerBuilder` to our set of DDC builders (enabled via the `web-hot-reload` config). This builder keeps a `PersistentFrontendServer` instance alive across rebuilds. Compile/recompile requests are queued via a `FrontendServerProxyDriver` resource.
* Uses `scratch_space` to record both 1) the main app entrypoint and 2) updated local files from the `entrypoint_marker` builder and the `module_builder` builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The `entrypoint_marker` builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request. 
* Requires that strongly connected components in both the frontend server and `build_runner` be disabled.

Test changes:
* Extends `build_test` to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds. 
* `build_test` doesn't support `runs_before` and other ordering rules in `build.yaml`, so the above changes allows a kind of imperative ordering, which is important for testing `entrypoint_marker`.

Minor changes:
* Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
* Added disposal logic to `scratch_space` so that rebuilds only retain modified files.
* Updated `scratch_space` `package_config.json` specs (`packageUri` and `rootUri`). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in `scratch_space`.
* Added `file` and `uuid` deps to `build_modules`.
* Moved around some helper functions.
* Ported some naming functions from the DDC runtime.

Author: Markzipan

build_modules/CHANGELOG.md

@@ -1,6 +1,11 @@
+## 5.1.0
+
+- Add drivers and state resources required for DDC + Frontend Server compilation.
+- Add an option to disable strongly connected components for determining module boundaries.
+
 ## 5.0.18
 
-- Remove unused dev depencency: `build_runner_core`.
+- Remove unused dev dependency: `build_runner_core`.
 - Allow Dart SDK 3.10.x and 3.11 prerelease.
 
 ## 5.0.17

build_modules/lib/build_modules.dart

@@ -2,17 +2,25 @@
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
+export 'src/ddc_names.dart';
 export 'src/errors.dart' show MissingModulesException, UnsupportedModules;
-export 'src/kernel_builder.dart'
-    show KernelBuilder, multiRootScheme, reportUnusedKernelInputs;
+export 'src/frontend_server_resources.dart'
+    show frontendServerState, frontendServerStateResource;
+export 'src/kernel_builder.dart' show KernelBuilder, reportUnusedKernelInputs;
 export 'src/meta_module_builder.dart'
     show MetaModuleBuilder, metaModuleExtension;
 export 'src/meta_module_clean_builder.dart'
     show MetaModuleCleanBuilder, metaModuleCleanExtension;
 export 'src/module_builder.dart' show ModuleBuilder, moduleExtension;
+export 'src/module_library.dart' show ModuleLibrary;
 export 'src/module_library_builder.dart'
     show ModuleLibraryBuilder, moduleLibraryExtension;
 export 'src/modules.dart';
 export 'src/platform.dart' show DartPlatform;
 export 'src/scratch_space.dart' show scratchSpace, scratchSpaceResource;
-export 'src/workers.dart' show dartdevkDriverResource, maxWorkersPerTask;
+export 'src/workers.dart'
+    show
+        dartdevkDriverResource,
+        frontendServerProxyDriverResource,
+        maxWorkersPerTask,
+        persistentFrontendServerResource;

build_modules/lib/src/common.dart

@@ -2,9 +2,17 @@
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
+import 'dart:io';
+
 import 'package:build/build.dart';
+import 'package:path/path.dart' as p;
 import 'package:scratch_space/scratch_space.dart';
 
+const multiRootScheme = 'org-dartlang-app';
+const webHotReloadOption = 'web-hot-reload';
+final sdkDir = p.dirname(p.dirname(Platform.resolvedExecutable));
+final packagesFilePath = p.join('.dart_tool', 'package_config.json');
+
 final defaultAnalysisOptionsId = AssetId(
   'build_modules',
   'lib/src/analysis_options.default.yaml',
@@ -16,6 +24,12 @@ String defaultAnalysisOptionsArg(ScratchSpace scratchSpace) =>
 enum ModuleStrategy { fine, coarse }
 
 ModuleStrategy moduleStrategy(BuilderOptions options) {
+  // DDC's Library Bundle module system only supports fine modules since it must
+  // align with the Frontend Server's library management scheme.
+  final usesWebHotReload = options.config[webHotReloadOption] as bool? ?? false;
+  if (usesWebHotReload) {
+    return ModuleStrategy.fine;
+  }
   final config = options.config['strategy'] as String? ?? 'coarse';
   switch (config) {
     case 'coarse':

build_web_compilers/lib/src/ddc_names.dart renamed to build_modules/lib/src/ddc_names.dart

@@ -1,14 +1,14 @@
-// Copyright (c) 2018, the Dart project authors.  Please see the AUTHORS file
+// Copyright (c) 2025, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 import 'package:path/path.dart' as p;
 
+/// Logic in this file must be synchronized with their namesakes in DDC at:
+/// pkg/dev_compiler/lib/src/compiler/js_names.dart
+
 /// Transforms a path to a valid JS identifier.
 ///
-/// This logic must be synchronized with [pathToJSIdentifier] in DDC at:
-/// pkg/dev_compiler/lib/src/compiler/module_builder.dart
-///
 /// For backwards compatibility, if this pattern is changed,
 /// dev_compiler_bootstrap.dart must be updated to accept both old and new
 /// patterns.
@@ -35,12 +35,11 @@ String toJSIdentifier(String name) {
   for (var i = 0; i < name.length; i++) {
     final ch = name[i];
     final needsEscape = ch == r'$' || _invalidCharInIdentifier.hasMatch(ch);
-    if (needsEscape && buffer == null) {
-      buffer = StringBuffer(name.substring(0, i));
-    }
-    if (buffer != null) {
-      buffer.write(needsEscape ? '\$${ch.codeUnits.join("")}' : ch);
+    if (needsEscape) {
+      buffer ??= StringBuffer(name.substring(0, i));
     }
+
+    buffer?.write(needsEscape ? '\$${ch.codeUnits.join("")}' : ch);
   }
 
   final result = buffer != null ? '$buffer' : name;
@@ -56,7 +55,11 @@ String toJSIdentifier(String name) {
 /// Also handles invalid variable names in strict mode, like "arguments".
 bool invalidVariableName(String keyword, {bool strictMode = true}) {
   switch (keyword) {
-    // http://www.ecma-international.org/ecma-262/6.0/#sec-future-reserved-words
+    // https://262.ecma-international.org/6.0/#sec-reserved-words
+    case 'true':
+    case 'false':
+    case 'null':
+    // https://262.ecma-international.org/6.0/#sec-keywords
     case 'await':
     case 'break':
     case 'case':
@@ -79,7 +82,6 @@ bool invalidVariableName(String keyword, {bool strictMode = true}) {
     case 'import':
     case 'in':
     case 'instanceof':
-    case 'let':
     case 'new':
     case 'return':
     case 'super':
@@ -99,6 +101,7 @@ bool invalidVariableName(String keyword, {bool strictMode = true}) {
     // http://www.ecma-international.org/ecma-262/6.0/#sec-identifiers-static-semantics-early-errors
     case 'implements':
     case 'interface':
+    case 'let':
     case 'package':
     case 'private':
     case 'protected':