Skip to content

fix(worker): deliver fiber activity cancellation via scheduler fiber_… - #519

Open
michaeldiscala wants to merge 1 commit into
temporalio:mainfrom
michaeldiscala:fix-fiber-activity-cancel-strands-worker
Open

fix(worker): deliver fiber activity cancellation via scheduler fiber_…#519
michaeldiscala wants to merge 1 commit into
temporalio:mainfrom
michaeldiscala:fix-fiber-activity-cancel-strands-worker

Conversation

@michaeldiscala

Copy link
Copy Markdown

…interrupt

Cancel callbacks run on the canceler's fiber, which for activity cancellation is the worker's single task-dispatch fiber. Fiber#raise uses resume semantics, so under a scheduler that drives fibers with Fiber#transfer (such as async) the target fiber never returns to its resumer and the dispatch fiber is suspended forever. The worker then stops dispatching all work -- activity tasks and workflow activations alike -- while the process and the reactor stay healthy.

Deliver the exception through Fiber::Scheduler#fiber_interrupt (Ruby 3.3+) when the scheduler provides one, so the raise is performed by the scheduler itself and the caller returns immediately. Schedulers without the hook keep the existing Fiber#raise behavior.

Fixes #518

What was changed

Moves from raising on a fiber to using the scheduler interrupt for CanceledErrors.

Why?

Prevents FIber based workers from stalling.

Checklist

  1. Closes [Bug] START_TO_CLOSE timeouts deterministically hang Fiber workers #518

  2. Tested against the reproduction harness on [Bug] START_TO_CLOSE timeouts deterministically hang Fiber workers #518 and confirmed that activities proceed as expected

  3. WARNING: Due to the nature of the bug, the new test case just hangs the entire test suite on the unpatched code - I didn't see an easy way to have it just go red quickly. On the patched code it runs as expected.

…interrupt

Cancel callbacks run on the canceler's fiber, which for activity cancellation is
the worker's single task-dispatch fiber. `Fiber#raise` uses resume semantics, so
under a scheduler that drives fibers with `Fiber#transfer` (such as `async`) the
target fiber never returns to its resumer and the dispatch fiber is suspended
forever. The worker then stops dispatching all work -- activity tasks and
workflow activations alike -- while the process and the reactor stay healthy.

Deliver the exception through `Fiber::Scheduler#fiber_interrupt` (Ruby 3.3+)
when the scheduler provides one, so the raise is performed by the scheduler
itself and the caller returns immediately. Schedulers without the hook keep the
existing `Fiber#raise` behavior.

Fixes temporalio#518
@michaeldiscala
michaeldiscala requested a review from a team as a code owner August 7, 2026 16:05
@CLAassistant

CLAassistant commented Aug 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@chris-olszewski chris-olszewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good with a few exceptions.

Could you add a test similar to test_activity_shielding, but it uses

class FiberShieldingActivity < ShieldingActivity
    activity_executor :fiber
end

It displays why we need the additional fiber != ::Fiber.current check before using fiber_interrupt

# worker's single task-dispatch fiber. Fiber#raise uses resume semantics, so under a
# scheduler that drives fibers with Fiber#transfer (e.g. async) the target never returns
# to its resumer and the dispatch fiber is suspended forever, wedging the whole worker.
# Schedulers expose fiber_interrupt (Ruby 3.3+) to perform the raise themselves, which

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I see, this just got added in Ruby 4

context&.cancellation&.add_cancel_callback do
fiber.raise(Error::CanceledError.new('Activity canceled'))
error = Error::CanceledError.new('Activity canceled')
if scheduler

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to use fiber_interrupt if we're on the same fiber already as it bypasses cancellation shielding.

Suggested change
if scheduler
if scheduler && fiber != ::Fiber.current

Comment on lines +42 to +47
# Cancel callbacks run on the canceler's fiber, which for activity cancellation is the
# worker's single task-dispatch fiber. Fiber#raise uses resume semantics, so under a
# scheduler that drives fibers with Fiber#transfer (e.g. async) the target never returns
# to its resumer and the dispatch fiber is suspended forever, wedging the whole worker.
# Schedulers expose fiber_interrupt (Ruby 3.3+) to perform the raise themselves, which
# returns to the caller immediately.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure if all of the words in the comment are useful.

Suggested change
# Cancel callbacks run on the canceler's fiber, which for activity cancellation is the
# worker's single task-dispatch fiber. Fiber#raise uses resume semantics, so under a
# scheduler that drives fibers with Fiber#transfer (e.g. async) the target never returns
# to its resumer and the dispatch fiber is suspended forever, wedging the whole worker.
# Schedulers expose fiber_interrupt (Ruby 3.3+) to perform the raise themselves, which
# returns to the caller immediately.
# Directly raising from another fiber can strand a `Fiber#transfer`-based
# scheduler's current fiber.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] START_TO_CLOSE timeouts deterministically hang Fiber workers

3 participants