Fix GrpcWorkerChannel.StartWorkerProcessAsync timeout #10937
+969
−1,367
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jviau
commented
Mar 21, 2025
| await _rpcWorkerProcess.StartProcessAsync(); | ||
| _state = _state | RpcWorkerChannelState.Initializing; | ||
| await _workerInitTask.Task; | ||
| await _rpcWorkerProcess.StartProcessAsync(cancellationToken); |
There was a problem hiding this comment.
This is the important change - we will now wait on either worker fully initialized (gRPC connection established) or worker exits (in which case, we will re-throw any failures the worker experience).
brettsam
reviewed
Mar 24, 2025
brettsam
reviewed
Mar 24, 2025
|
/azp run host.integration-tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run host.public |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@jviau @mattchenderson I was just investigating this yesterday! Turns out this is the same reason that we are stuck for 60s when trying to exit the CLI during startup (Azure/azure-functions-core-tools#4355). Glad to see there is a already a fix before I opened one :) I'm going to test your changes out with core tools and see if it addresses the issue like I expect. Looks like it works? Before this change, it would hang for 60 seconds waiting on the timeout from PendingItem. For my version of the fix, I was considering passing a CT to PendingItem and registering an edit: more logs - this is outside of my debug session - it takes 5 seconds to shutdown after we cancel: For core tools at least, I would love to reduce this time but I can probably just inject options in the webhost to reduce the flush logs timer to a 1 second or something like that. Anywho not related to this PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue describing the changes in this PR
resolves #issue_for_this_pr
Pull request checklist
IMPORTANT: Currently, changes must be backported to the
in-procbranch to be included in Core Tools and non-Flex deployments.in-procbranch is not requiredrelease_notes.md-- TODOAdditional information
This PR improves the ScriptHost startup experience with a bad worker. Today, if a worker crashes or exits immediately after startup, then the
GrpcWorkerChannel.StartWorkerProcessAsyncwill block on_workerInitTask.Taskuntil it eventually times out. This tends to fault the entire host (at least during debugging).To address this, a
WorkerProcess.WaitForExitAsyncis added andGrpcWorkerChannel.StartWorkerProcessAsyncwill also wait on that, improving the responsiveness to a worker exiting before connecting gRPC events.