ops: Actions stopped registering PR-triggered runs 2026-08-06 ~11:38 UTC — scheduled jobs hang into their timeouts, no required context can report

[macanderson]

Problem

GitHub Actions has stopped picking up PR-triggered workflow runs for this repository. Observed 2026-08-06 ~18:20 UTC:

• The newest ci.yml run of any kind is 31098058151 (pull_request, 11:38 UTC). No PR-triggered run has registered since, on any workflow.
• PR ci: make the required contexts always report — path filters move from the trigger into the jobs (#1892) #1897 (pushed ~18:05 UTC) has zero GitHub Actions check-runs on its head SHA after 15+ minutes — only the external apps (Vercel, Sourcery) reported. gh run list --branch fix/1892-required-contexts-always-report returns nothing.
• Scheduled runs are hanging into their timeout-minutes instead of executing: automerge-nudge 31117785417 (15:53 UTC) and 31124388802 (17:52 UTC), and release-reconcile 31124722611 (17:59 UTC) each ran ~15m and ended failure with the single job cancelled and no failed step — the signature of a job that never really got a healthy runner, not of a code failure. The same workflows succeeded in seconds earlier the same day (13:22, 15:26 UTC).

Impact

• Every open PR is unverifiable: the required contexts (fmt + clippy + test, cargo deny + cargo audit, harbor_adapter + analyzer pytest) cannot report, so nothing can merge without an admin bypass.
• ci: make the required contexts always report — path filters move from the trigger into the jobs (#1892) #1897 (the fix for CI: a docs-only PR can never merge — required contexts never report under ci.yml's paths-ignore, and the merge queue that made that safe is off #1892) is itself waiting on this to demonstrate its rust=true path.

How to verify / what "done" looks like

1. gh run list --repo macanderson/stella --limit 5 shows fresh pull_request runs again (push any branch or re-run a PR head to test).
2. gh pr checks 1897 shows the is this diff prose-only? / fmt + clippy + test / cargo deny + cargo audit contexts.
3. Scheduled automerge-nudge / release-reconcile complete in seconds again.

Likely cause / constraints

An org-level Actions availability problem (runner starvation or a billing hold — the latter has happened to this org before; see RELEASING.md's local-release path, which AGENTS.md names as the fallback when Actions is unavailable entirely). Needs an org admin to check Settings → Billing and the Actions status; nothing in-repo can fix it. If it self-resolves, close this with the timestamp it recovered so the gap in run history is explained.

Refs #1897 #1892 #1645

[macanderson]

Root cause identified (2026-08-06 18:45 UTC): GitHub platform outage, not org-side starvation or billing. githubstatus.com reports Actions: major_outage (Pages too; overall 'Partial System Outage'). Evidence: hosted-runner jobs fail with 'The job was not acquired by Runner of type hosted even after multiple attempts' (run 31124388802), scheduled workflows time out at acquisition, and no PR-triggered run has registered since ~11:38 UTC. The repo is public, so paid-minutes exhaustion is not in play. Nothing in-repo or org-side to fix — this closes when GitHub recovers and a PR-triggered run is observed completing. Re-runs may be needed for workflows that failed at acquisition during the window.

[macanderson]

Status update — the symptom changed, but merges are still blocked.

Actions is registering pull_request-triggered runs again (ci, msrv, docs-guards, dependency-review all created), so the original "no runs register at all" symptom is gone.

They do not start. Every run created in the last hour — pushes and PRs alike — is still queued, none has moved to in_progress, and the oldest has been queued 35+ minutes. One docs-guards run did complete success, so the queue is not entirely dead, just starved.

Net effect on the backlog is unchanged: required contexts (fmt + clippy + test, cargo deny + cargo audit, harbor_adapter + analyzer pytest) cannot report, so no PR can reach a mergeable state regardless of its contents. Six PRs are currently open and locally green behind this.

This needs an operator with billing/runner access — it is not fixable from a PR.

[macanderson]

A defect has now reached main through this outage — and a full local gate audit of what else did.

It is no longer only blocking merges

PR #1920 merged on absent required contexts (every ci.yml run since 19:03 is still queued) and put a doc-warnings failure on main:

error: could not document `stella-cli`
  --> crates/stella-cli/src/daemon/boot.rs:130
   |     /// ([`supervised::APPROVAL_REQUEST`]).
   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no item named `supervised` in scope

That one is mine and PR #1929 fixes it. The point for this issue is the mechanism: branch protection cannot distinguish "the required check passed" from "the required check never ran" when the run is stuck in queued. This is a different failure from #1645's — not a red check overridden, but no check at all — and it means the outage is now actively admitting defects rather than merely holding PRs.

What else slipped through: nothing

I ran the entire gate locally against origin/main to find out.

Tier	Result
20 toolchain-free guards (make guards-fast)	OK
cargo fmt --check	OK
wire-schema	OK
clippy --workspace --all-targets -D warnings	OK
cargo test --workspace	OK — 123 test binaries, 0 failures
rustdoc -D warnings	FAILS (PR #1929)

So main is one merge away from a fully green gate. That is the reassuring half.

Current queue state

gh run list --workflow=ci.yml shows every run since 19:03 in queued — none has reached in_progress. The only checks reporting on PRs are third-party (Vercel, Sourcery), which are not among the three required contexts (fmt + clippy + test, cargo deny + cargo audit, harbor_adapter + analyzer pytest).

This needs an operator with billing/runner access. Until then, treat every merge as unchecked — running make gate locally before merging is the only thing standing in for CI, and it is what caught the break above.