feat: schedule Slurm jobs with weighted runner leases

[cquil11]

Summary

This proof of concept adds exact weighted GitHub-side admission for Slurm allocations.

• Generates node-count for every multi-node matrix row from an explicit aggregate num-nodes, checked-in recipe resources, explicit PREFILL_NODES/DECODE_NODES, or worker GPU topology.
• Allows num-nodes only for multinode: true, disagg: false; disaggregated entries reject it because their separate prefill/decode allocations determine the total.
• Adds nodes:N to queued jobs when both priority scheduling and NODE_SLOT_SCHEDULER_ENABLED=true; single-node jobs request nodes:1.
• Uses nodes:N as request metadata rather than a permanent runner capability.
• Uses node count as a secondary priority tie-breaker so otherwise-equal smaller allocations run first.
• Keeps the feature disabled by default.

Runtime allocation is implemented by the existing priority controller in the companion draft PR: inferencex-ci-tracker#7.

The existing aggregate master-config entries are backfilled in the native stacked PR above this one: #2479.

Lease lifecycle

For a job requesting nodes:3, the controller:

1. Finds three online, unleased runner tokens in one compatible cluster:* pool.
2. Adds one shared ci-lease-* label to all three runners.
3. Re-reads and verifies the complete lease.
4. Adds the queued job unique ci-job-*, ci-attempt-*, and nodes:3 labels to one anchor runner.
5. Preserves the lease while the anchor is queued or busy.
6. Removes the lease after completion or during stale-state reconciliation.

The unique ci-job-* label remains the dispatch gate. The other leased runners stay idle as abstract capacity tokens for the Slurm allocation.

On an 18-node pool, a running 10-node lease leaves eight tokens. A queued nine-node job remains blocked, while an eight-node job can backfill the exact remainder. Therefore managed work satisfies:

sum(admitted node counts) <= online cluster capacity

Slurm remains the final authority for external users, offline compute nodes, reservations, and jobs submitted outside this managed path.

Activation

1. Merge and deploy inferencex-ci-tracker#7.
2. Verify managed runners have exactly one correct cluster:* label. Unclustered legacy runners remain eligible only for one-node jobs.
3. Drain previously queued/running multi-node jobs so no pre-feature job is accounted as one node.
4. Set NODE_SLOT_SCHEDULER_ENABLED=true. PRIORITY_SCHEDULER_ENABLED must also remain enabled.

No permanent nodes:N label plan is required. The controller removes old static nodes:* labels as it reconciles idle runners.

Validation

• python -m pytest utils/matrix_logic/ utils/test_ci_priority.py -q — 253 passed.
• Generated all NVIDIA and AMD master-config rows; every multi-node row has node-count.
• Compared 216 generated rows across 185 checked-in srt-slurm recipes with authoritative resource totals; zero mismatches.
• actionlint found no new workflow-call or node-lease expression errors; existing unrelated shell/style warnings remain.
• Companion controller: 22 focused tests pass, including 18-node 10 + 9, exact 10 + 8 backfill, running-lease preservation, invariant coverage for pool sizes 1–20, legacy-runner fallback, and partial-write rollback.
• Changed controller sources pass focused TypeScript checking and ESLint.

[claude]

Claude finished @cquil11's task in 8m 4s —— View job

---

Review — feat: schedule Slurm jobs with weighted runner leases

• Gather context (read AGENTS.md, PR diff)
• Review changed files for bugs/logic errors
• Line count report for generate_sweep_configs.py
• Post inline comments and summary

Summary: One 🟡 warning — an empty node-count (e.g. trusted changelog dispatch re-benchmarking a pre-feature inputs.ref, whose checkout generates rows without the key) formats an unsatisfiable nodes: label into runs-on when NODE_SLOT_SCHEDULER_ENABLED=true, hanging the job until timeout; suggested a one-line guard in benchmark-multinode-tmpl.yml. Everything else verified clean: all 7 multinode template call sites pass node-count (the run-sweep 8k1k site is covered by the *multi-node-inputs YAML anchor), the nested runs-on expression is balanced with correct &&/|| semantics, the new node-count tests are arithmetically consistent with the fixtures and the kimi recipe (3+4=7), the −0.001/node tie-breaker can't cross the smallest real priority gap (0.25) below 251 nodes, and process_changelog.py round-trips node-count through the new pydantic fields. Note: I could not run pytest in this sandbox (command approval denied), so test results rely on static verification plus the author's reported 246-pass run.

[claude]

📊 Line Count Report

• Total Lines: 1525
• Base Lines: 1383
• Change: 📈 +142 lines