Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure

[Markzipan]

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:

1. A persistently running frontend server is initialized once when a build starts.
2. build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
3. When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
4. The frontend server processes compilation requests and serves compiled JS files back to build_runner (hot-reload ready).

Major changes:

• Adds a DdcFrontendServerBuilder to our set of DDC builders (enabled via the web-hot-reload config). This builder keeps a PersistentFrontendServer instance alive across rebuilds. Compile/recompile requests are queued via a FrontendServerProxyDriver resource.
• Uses scratch_space to record both 1) the main app entrypoint and 2) updated local files from the entrypoint_marker builder and the module_builder builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The entrypoint_marker builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request.
• Requires that strongly connected components in both the frontend server and build_runner be disabled.

Test changes:

• Extends build_test to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds.
• build_test doesn't support runs_before and other ordering rules in build.yaml, so the above changes allows a kind of imperative ordering, which is important for testing entrypoint_marker.

Minor changes:

• Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
• Added disposal logic to scratch_space so that rebuilds only retain modified files.
• Updated scratch_space package_config.json specs (packageUri and rootUri). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in scratch_space.
• Added file and uuid deps to build_modules.
• Moved around some helper functions.
• Ported some naming functions from the DDC runtime.

Currently doesn't support live-reloading (functionality appears to have been broken a while ago). This'll be added in an upcoming change and permit webdev-like auto-hot-reload on save (on top of manual).

Enable this by adding the following to a project's build.yaml:

global_options:
  build_web_compilers|sdk_js:
    options:
      web-hot-reload: true
  build_web_compilers|entrypoint:
    options:
      web-hot-reload: true
  build_web_compilers|ddc:
    options:
      web-hot-reload: true
  build_web_compilers|ddc_modules:
    options:
      web-hot-reload: true

[github-actions]

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

[biggs0125]

Requires that strongly connected components in both the frontend server and build_runner be disabled.

Can you say more about the affect this has on the structure of compilations? Are we doing more work by turning off SCCs? My assumption was that SCCs allowed us to limit the scope of an invalidation.

[Markzipan]

@davidmorgan Thanks! It's a huge change, so please take your time. I wasn't able to configure it in a prettier way since I'm flying out for two weeks.

I'm highly unopinionated wrt testing, but it was definitely a struggle to get testBuilders to cooperate with rebuilds. Please send any followup recs for better ways to represent this test. The important bits to me are: 1) respecting builder order (as per build.yaml) since that matters now, 2) being able to run rebuilds vs new builds (or determine build boundaries), 3) updating files passed to the builders.

@biggs0125 The SCCs here operate on a the import-graph level. DDC's old module system requires that cyclic dependencies be 'unified' into a single module. Both the Frontend Server and build_runner doing this non-deterministically might cause a <--> b to be merged into a in build_runner and b in the Frontend Server, which gives us "import not found" errors.

With SCCs enabled, a.dart would invoke a single builder on module(a, b). Disabling them, two builders, module(a) and module(b), are invoked. The net effect of this on some apps I've tested hasn't been large, as most modules don't end up in an SCC anyway. The largest ramifications would be a hypothetical gigantic app that wants to bundle smaller JS files. But that would ideally be specified statically, so we'd be able to pass it consistently to both build_runner and the Frontend Server. The particularly annoying bit about SCCs is that both systems maintain their own independent algorithm without any configurability. I alternatively could've 1) poked a hole into the Frontend Server to allow per-compile module-to-lib maps and 2) had build_runner serialize its module-to-lib maps and send that per-compile to the Frontend Server. But I figured that was too much of a mess.

Re: failures. I think some tests are failing due to API changes without pinning new versions? I'll add those (maybe in a separate PR) if the current approach looks sound.

[davidmorgan]

This generally looks good, a few comments re: scratch_space.

Re: testing :)

You sort of don't need to pass assetGraph, it's serialized to the readerWriter and the next build will read it if it's there.

But, the next build will discard the asset graph if any builders changed, so you have to pass the exact same builders each time, which means you will get a full build the first time you call it anyway.

You commented with build.yaml ordering is not respected, and that's true. Adding workarounds for that is not too pretty, it's getting very close to a real build but now with quite a lot of configuration that doesn't exactly match a real build.

Would you be up for trying a different way of testing that is a real build?

I added tests recently that make it easy to set up some packages, run build_runner for real and check the output. Since you want persistent processes, watch mode would make sense, so something like:

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 9f07c7e

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file

There is an example web build using the same test infra here

https://github.com/dart-lang/build/blob/master/build_runner/test/integration_tests/web_compilers_test.dart

The test infra is currently internal to build_runner, mainly build_runner/test/common/build_runner_tester.dart, if it looks useful let's just hack around that for now and I'll work out how best to clean it up, I guess adding to build_test is one option.

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:build	4.0.2	already published at pub.dev
package:build_config	1.2.0	already published at pub.dev
package:build_daemon	4.1.0	already published at pub.dev
package:build_modules	5.1.0	ready to publish	build_modules-v5.1.0
package:build_runner	2.10.2-wip	WIP (no publish necessary)
package:build_test	3.5.2-wip	WIP (no publish necessary)
package:build_web_compilers	4.4.0	ready to publish	build_web_compilers-v4.4.0
package:scratch_space	1.2.0	ready to publish	scratch_space-v1.2.0

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[Markzipan]

@davidmorgan Wow, the tests in build/build_runner/test/integration_tests are perfect for this. I've replaced the build_web_compilers FES driver test with additional web_compilers_test.dart tests, added a separate resource for FES-specific state, and bumped scratch_space's ver.

Thanks for the review - back from vacation!

[davidmorgan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

[davidmorgan]

Welcome back!

Looking good so far.

There are some CI failures, could you take a look at those please?

[davidmorgan]

Hmmmm multiple expectations is a bit awkward, I wonder if there's a nicer way.

Would it work to accumulate lines and return them, then the second expectation can be checked on the result?

Looking at how this method is used, I think maybe something like this:

• most callers are not using the result, so how about the base method is Future<void> expect.
• a few callers are using the single matching line, so how about Future<String> expectAndGetLine for that
• and then I think you can do what you want with the full accumulated output until the match, how about Future<List<String>> expectAndGetBlock

?

[Markzipan]

I like that change, but I the newly added tests' reqs are:

1. we should be able to expect that a test throws
2. we should be able to match on more complex patterns (more than one line of match)

The problem with returning a block is that expect waits on a stream until it times out for 30s or encounters pattern, so we don't have a good way to determine when a block is finished accumulating. Really it should just accept a List<Matcher> and return if all matchers are successful. Thoughts?

[davidmorgan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

Whoops, looks like I had unpublished comments that you addressed by coincidence. Sorry about that! Now published.

[Markzipan]

It looks like you made some changes in response to my comments but I don't see any responses to the comments so I'm not sure if you're done--please let me know if/when I should take another look. Thanks!

@davidmorgan Oh no - you didn't see my comments? I can ping you when things are fully ready, but I've been in and out since I'm sick with something mysterious lol

@nshahan Thanks for reviewing! I updated some of the formatting based on your feedback.

It looks like I've encountered a windows pathing issue in some SDK lib. I'll investigate tomorrow :(