We are building a per-job ephemeral runner controller and need guaranteed stickiness between a job and a specific agent (each job must boot exactly one VM, run on it, and tear it down). To achieve this, we are experimenting with creating a dedicated scale set per job on demand — each scale set is created when a job enters our queue, so that the JobAssigned message arrives on the scale set that corresponds to that exact job.
The problem we are running into: jobs dispatched by GitHub carry user-defined labels. Since our per-job scale sets all use the same label set derived from the job's runs-on, there is no guarantee that a JobAssigned for job A arrives on scale set A rather than scale set B, which was created for a different job with the same labels.
We ran a PoC with two scale sets (runhub-poc-test and runhub-poc-test2) sharing identical labels and observed the following:
Core question — job distribution across multiple same-label scale sets
If we have N scale sets with identical labels and N jobs are dispatched simultaneously (all requiring those labels), how does GitHub route jobs to scale sets? Is there any balancing strategy (round-robin, random, FIFO by creation time, etc.)? Or is routing arbitrary from the controller's perspective?
This is the most important question for us: we need to know whether it is safe to assume that each scale set gets exactly one distinct job, or whether multiple jobs can pile up on one scale set while others stay idle.
Observation 1 — No JobAvailable, direct JobAssigned
Even with two competing sessions listening on the same labels, GitHub never sent a JobAvailable message. One session received JobAssigned directly; the other received nothing. We expected the JobAvailable → AcquireJobs competitive round to fire.
- Is
JobAvailable only sent when maxCapacity > 1 or under specific broker conditions?
- Is direct
JobAssigned (skipping JobAvailable) the normal path for single-capacity scale sets?
Observation 2 — runnerRequestId: 0 in JobAssigned
The JobAssigned message had runnerRequestId: 0. Since AcquireJobs takes a list of runnerRequestIds, calling it after a direct JobAssigned is a no-op.
- Is
runnerRequestId always 0 in JobAssigned when the JobAvailable step was skipped?
- Is
AcquireJobs strictly a JobAvailable-phase API and irrelevant after JobAssigned?
Observation 3 — Routing with two competing sessions
With two sessions open on same-label scale sets, one session received JobAssigned, the other received nothing.
- Is the losing session expected to stay idle indefinitely, or will it eventually receive a cancellation/timeout event?
- Is there any signal a session can use to know it "lost" and should tear down?
More broadly: is there any mechanism in the scale set API that would allow guaranteed job-to-scale-set routing? For example, a way to scope a scale set so it only receives a specific job ID, or a way to reject/NACK a JobAssigned so it gets rerouted to a different scale set?
We are building a per-job ephemeral runner controller and need guaranteed stickiness between a job and a specific agent (each job must boot exactly one VM, run on it, and tear it down). To achieve this, we are experimenting with creating a dedicated scale set per job on demand — each scale set is created when a job enters our queue, so that the
JobAssignedmessage arrives on the scale set that corresponds to that exact job.The problem we are running into: jobs dispatched by GitHub carry user-defined labels. Since our per-job scale sets all use the same label set derived from the job's
runs-on, there is no guarantee that aJobAssignedfor job A arrives on scale set A rather than scale set B, which was created for a different job with the same labels.We ran a PoC with two scale sets (
runhub-poc-testandrunhub-poc-test2) sharing identical labels and observed the following:Core question — job distribution across multiple same-label scale sets
If we have N scale sets with identical labels and N jobs are dispatched simultaneously (all requiring those labels), how does GitHub route jobs to scale sets? Is there any balancing strategy (round-robin, random, FIFO by creation time, etc.)? Or is routing arbitrary from the controller's perspective?
This is the most important question for us: we need to know whether it is safe to assume that each scale set gets exactly one distinct job, or whether multiple jobs can pile up on one scale set while others stay idle.
Observation 1 — No
JobAvailable, directJobAssignedEven with two competing sessions listening on the same labels, GitHub never sent a
JobAvailablemessage. One session receivedJobAssigneddirectly; the other received nothing. We expected theJobAvailable→AcquireJobscompetitive round to fire.JobAvailableonly sent whenmaxCapacity > 1or under specific broker conditions?JobAssigned(skippingJobAvailable) the normal path for single-capacity scale sets?Observation 2 —
runnerRequestId: 0inJobAssignedThe
JobAssignedmessage hadrunnerRequestId: 0. SinceAcquireJobstakes a list ofrunnerRequestIds, calling it after a directJobAssignedis a no-op.runnerRequestIdalways0inJobAssignedwhen theJobAvailablestep was skipped?AcquireJobsstrictly aJobAvailable-phase API and irrelevant afterJobAssigned?Observation 3 — Routing with two competing sessions
With two sessions open on same-label scale sets, one session received
JobAssigned, the other received nothing.More broadly: is there any mechanism in the scale set API that would allow guaranteed job-to-scale-set routing? For example, a way to scope a scale set so it only receives a specific job ID, or a way to reject/NACK a
JobAssignedso it gets rerouted to a different scale set?