Implement changes needed in the Host to decouple workers from the Host release #11111

surgupta-msft · 2025-06-09T19:38:46Z

Issue describing the changes in this PR

resolves #10944

Update - working with Brett on addressing the comment - #11111 (comment).

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

Backporting to the in-proc branch is not required
- Otherwise: Link to backporting PR
My changes do not require documentation changes
- Otherwise: Documentation issue linked to PR
My changes should not be added to the release notes for the next release
- Otherwise: I've added my notes to release_notes.md
My changes do not need to be backported to a previous version
- Otherwise: Backport tracked by issue/PR #issue_or_pr
My changes do not require diagnostic events changes
- Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

This pull request introduces changes to decouple language workers from the Host release and enable dynamic worker resolution. Key updates include new worker configuration resolvers, environment enhancements, and feature flags to control worker behavior.

Backlog issues - Link
Design doc - Link

Flows covered in this PR -

LanguageWorkerOptionsSetup -
- Uses IWorkerConfigurationResolverFactory to handle the instantiation logic of the resolver depending on if the feature is enabled.
- Creates and sends an instance of IWorkerConfigurationResolver when creating an instance of RPCWorkerConfigFactory.
RPCWorkerConfigFactory -
- Calls GetWorkerConfigs() method of IWorkerConfigurationResolver to get the required worker configs.
- Moved some methods to WorkerConfigurationHelper.cs to enable reusing of workers profile evaluation logic.
Resolving Worker Configurations -
- Added DefaultWorkerConfigurationResolver to scan the Host's "workers" directory for worker configurations.
- Introduced DynamicWorkerConfigurationResolver, which dynamically resolves worker configurations based on -
  - Environment settings and probing paths
  - Host-Worker compatibility check
  - Leveraging Release channel concept to point to a previous version of the worker
  - Fallback to the old flow if worker is not found at the probing path level
Environment Updates -
- Added IsWindowsEnvironment() to determine the windows environment type.
- Added IsDynamicWorkerResolutionEnabled method to EnvironmentExtensions class to enable/disable dynamic worker resolution using feature flag and hosting configuration.
- Added a new feature flag FeatureFlagDisableWorkerProbingPaths to disable dynamic resolution feature by users.

Note: The decoupling workers flow is disabled by default in this PR. We will enable the flow after completing other relevant backlog items which will be included in follow-up PRs.

surgupta-msft · 2025-06-09T21:42:36Z

src/WebJobs.Script/Workers/Rpc/Configuration/RpcWorkerConfigFactory.cs

@@ -144,30 +176,16 @@ internal void AddProvider(string workerDir)

                    _logger.LogDebug("Found worker config: {workerConfigPath}", workerConfigPath);

-                    var workerConfig = GetWorkerConfigJsonElement(workerConfigPath);


No change in logic for this line. Just moved the method GetWorkerConfigJsonElement() to class WorkerConfigurationHelper

surgupta-msft · 2025-06-09T21:43:50Z

src/WebJobs.Script/Workers/Rpc/Configuration/RpcWorkerConfigFactory.cs

@@ -144,30 +176,16 @@ internal void AddProvider(string workerDir)

                    _logger.LogDebug("Found worker config: {workerConfigPath}", workerConfigPath);

-                    var workerConfig = GetWorkerConfigJsonElement(workerConfigPath);
-                    var workerDescriptionElement = workerConfig.GetProperty(WorkerConstants.WorkerDescription);


All these lines (148-170) are moved to a method WorkerConfigurationHelper.GetWorkerDescription() - https://gist.github.com/surgupta-msft/b416af7ca3c8794849832cc56f7a11de/revisions

surgupta-msft · 2025-06-09T21:45:29Z

src/WebJobs.Script/Workers/Rpc/Configuration/RpcWorkerConfigFactory.cs

@@ -213,61 +231,6 @@ internal void AddProvider(string workerDir)
            }
        }

-        private static JsonElement GetWorkerConfigJsonElement(string workerConfigPath)


These methods are also moved to class WorkerConfigurationHelper as we will need to call these methods from class WorkerConfigurationResolver as part of compatibility check

surgupta-msft · 2025-06-09T21:46:18Z

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationHelper.cs

+namespace Microsoft.Azure.WebJobs.Script.Workers.Rpc.Configuration
+{
+    internal static class WorkerConfigurationHelper
+    {


All the methods in this class are taken from RPCWorkerConfigFactory with minor adjustments. See gist - https://gist.github.com/surgupta-msft/7eee560448e21ad1d450eb3e425895cd/revisions

…ling-workers

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

src/WebJobs.Script/Environment/EnvironmentExtensions.cs

src/WebJobs.Script/Config/FunctionsHostingConfigOptions.cs

src/WebJobs.Script/Workers/Rpc/RpcWorkerConstants.cs

src/WebJobs.Script/Workers/Rpc/IWorkerConfigurationResolver.cs

src/WebJobs.Script/Utility.cs

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverFactory.cs

test/TestWorkers/Probingpaths/workers/node/3.10.1/worker.config.json

src/WebJobs.Script/Workers/Rpc/Configuration/LanguageWorkerOptionsSetup.cs

liliankasem · 2025-07-08T00:16:13Z

Minor comment on branch name, we should use feature/ for feature branches as we have rules and CI setup for it (its protected) i.e. feature/decouple-workers

brettsam · 2025-07-08T18:51:52Z

src/WebJobs.Script/Workers/Rpc/Configuration/DefaultWorkerConfigurationResolver.cs

+        private readonly ILogger _logger;
+        private readonly IConfiguration _configuration;
+
+        public DefaultWorkerConfigurationResolver(IConfiguration configuration, ILogger logger)


I missed some of this in the first pass -- but we're trying to move away from passing in IConfiguration or IEnvironment to any classes. They aren't very descriptive -- like right here you have no idea as a consumer what part of IConfiguration is needed.

Instead, using some kind of IOptions<> object that builds the options (using IConfiguration) before passing them in is a much nicer contract and much easier to test/read.

Highly recommend moving away from this where you can. I'll call out some others as I see them.

If it works, I would prefer to address this issue in a separate PR. I have created an issue - #11185 for it and added it to the "Decoupling workers" milestone. This PR already contains a lot of logic and tests and it is also against a feature branch, so I am not merging changes into dev.

However, if you are really concerned about it, I can raise another PR with the fix targeting this PR branch (surgupta/decoupling-workers). We can merge that PR first before completing this one. Though it is not my preferred approach, as this PR is becoming long-standing and we are already using this pattern in multiple places in Host code, but I can do it if necessary.

Discussed offline. Will be implementing the improvement first.

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

kshyju

Let's also add the placeholder + specialization use case test to validate that flow.

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationHelper.cs

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverFactory.cs

src/WebJobs.Script/Workers/Rpc/IWorkerConfigurationResolver.cs

src/WebJobs.Script/Workers/Rpc/Configuration/DefaultWorkerConfigurationResolver.cs

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

src/WebJobs.Script/Workers/Rpc/Configuration/LanguageWorkerOptionsSetup.cs

surgupta-msft · 2025-07-15T03:26:13Z

Let's also add the placeholder + specialization use case test to validate that flow.

Yes, this will definitely be included with Integration and E2E tests changes. I am currently working on it - #11140

surgupta-msft added 20 commits May 25, 2025 18:02

Changes for Decoupling workers implementation

987fad0

Added test data for unit tests

734b029

Code cleanup and adding comments

cefdb92

Code refactoring to check probing paths enabled only once

021ac3e

Code refactoring

36abb61

Tests cleanup

8929697

Added test for LanguageWorkerOptionsSetup

1588711

Adding Tests for Utility class

2606f37

Adding more tests

08850c9

Changes in Tests

10d2e52

Updating LanguageWorkerOptionsSetupTests

95af17e

Minor refactoring and code cleanup

150aaf3

Fixing build warning

0def600

Fixing build issue in tests

cc3e044

Unit test fixing

26e8580

Stylecop fixes

9b6a945

Merge branch 'dev' into surgupta/decoupling-workers

e90ddda

Test data cleanup

ee60ec4

Test data cleanup

1495f31

Merge branch 'dev' into surgupta/decoupling-workers

c2628bb

surgupta-msft requested a review from a team as a code owner June 9, 2025 19:38

surgupta-msft commented Jun 9, 2025

View reviewed changes

Updated release notes

4ee497b

surgupta-msft changed the base branch from dev to surgupta/decouple-workers-feature June 11, 2025 05:39

surgupta-msft added 3 commits June 11, 2025 23:42

Merge branch 'surgupta/decouple-workers-feature' into surgupta/decoup…

c85ced3

…ling-workers

Renaming feature name

5168090

Updated tests

1ad3d14

Code cleanup and added more comments

9e4b960

satvu reviewed Jul 3, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs Outdated Show resolved Hide resolved

satvu reviewed Jul 3, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs Outdated Show resolved Hide resolved

satvu reviewed Jul 3, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs Outdated Show resolved Hide resolved

liliankasem reviewed Jul 3, 2025

View reviewed changes

surgupta-msft added 6 commits July 3, 2025 14:08

Adding condition if worker.config is empty

2585946

Addressed PR feedback and code refactoring

3c55d1d

Creating WorkerConfigurationResolverFactory

c5a1b0f

code cleanup

6449065

Minor code refactoring

5ad1699

Code cleanup and addressing PR feedback

d15984c

liliankasem reviewed Jul 7, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverFactory.cs Outdated Show resolved Hide resolved

liliankasem reviewed Jul 7, 2025

View reviewed changes

test/TestWorkers/Probingpaths/workers/node/3.10.1/worker.config.json Outdated Show resolved Hide resolved

liliankasem reviewed Jul 7, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/LanguageWorkerOptionsSetup.cs Show resolved Hide resolved

liliankasem approved these changes Jul 8, 2025

View reviewed changes

surgupta-msft added 2 commits July 7, 2025 20:14

Test folder update

c6b9e93

Test folder name update

be527c4

surgupta-msft changed the base branch from surgupta/decouple-workers-feature to feature/decouple-workers July 8, 2025 01:16

Merge with main

aaf5243

brettsam reviewed Jul 8, 2025

View reviewed changes

Added some code comments

e0cb000

kshyju reviewed Jul 9, 2025

View reviewed changes

surgupta-msft added 4 commits July 8, 2025 23:23

Addressing PR feedback

738cea9

Log improvement

710ae35

Condition update to include placeholder mode

90e0ad2

Merge with main

b3a4045

surgupta-msft added 2 commits July 15, 2025 01:04

Test fix

fbb3c10

Merge with main

2936fa8

		@@ -144,30 +176,16 @@ internal void AddProvider(string workerDir)

		_logger.LogDebug("Found worker config: {workerConfigPath}", workerConfigPath);

		var workerConfig = GetWorkerConfigJsonElement(workerConfigPath);

Implement changes needed in the Host to decouple workers from the Host release #11111

Are you sure you want to change the base?

Implement changes needed in the Host to decouple workers from the Host release #11111

Conversation

surgupta-msft commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue describing the changes in this PR

Pull request checklist

Additional information

Uh oh!

surgupta-msft Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liliankasem commented Jul 8, 2025

Uh oh!

brettsam Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kshyju left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

surgupta-msft commented Jul 15, 2025

Uh oh!

Uh oh!

surgupta-msft commented Jun 9, 2025 •

edited

Loading

surgupta-msft Jun 9, 2025 •

edited

Loading

surgupta-msft Jun 9, 2025 •

edited

Loading

surgupta-msft Jun 9, 2025 •

edited

Loading