fix(identity-jwt): JWKS refresh task is cancelled at startup and never runs

[terylt]

Description

The JWKS background refresh task is spawned onto a runtime that is dropped moments later, so it is cancelled before it performs a single refresh. JWKS keys are frozen at whatever was fetched during startup, for the life of the process.

Confirmed in both repos.

PPE side. identity-jwt's initialize() (builtins/plugins/identity-jwt/src/resolver.rs:295) spawns a ticker per JWKS issuer that declares a refresh_interval:

// resolver.rs:360
let handle = tokio::spawn(async move {
    let mut ticker = tokio::time::interval(interval);
    ticker.tick().await;          // skip the immediate first tick
    loop {
        ticker.tick().await;
        match source.build_async().await { ... }   // re-fetch, swap store
    }
});

tokio::spawn binds the task to whichever runtime is current when it is called.

Praxis side. PolicyFilter::new drives initialize() on a throwaway runtime, because the filter factory signature is sync (filter/src/builtins/http/security/policy/filter.rs:175):

let init: Result<(), String> = std::thread::spawn(move || -> Result<(), String> {
    let rt = tokio::runtime::Builder::new_current_thread().enable_all().build()?;
    rt.block_on(async move {
        tokio::time::timeout(init_timeout, mgr_for_init.initialize()).await
        ...
    })
})
.join()

rt is a local binding. It drops when the closure returns, immediately after initialize() completes. Dropping a tokio runtime cancels its tasks; they do not migrate.

Because the ticker deliberately skips its first immediate tick and fires at now + interval, it is cancelled having never refreshed once. This is not "refreshes stop early." It is zero refreshes, ever.

The abort-on-Drop bookkeeping at resolver.rs:262 and resolver.rs:402 is correct code doing nothing, because the tasks were already cancelled.

Impact

Two failure modes, both permanent until a gateway restart. The resolver's own comments (resolver.rs:482) name both and say refresh is what recovers them.

Key rotation never recovers. When the IdP rolls its signing keys, every token signed with the new key returns auth.unknown_kid. Identity resolution fails, identity failure is fail-closed, so the issuer's users are denied. Comment: "the IdP rolled and our refresh hasn't yet pulled the new key."

Soft-fail-at-boot has become permanent-fail-at-boot. This is the worse one. The design deliberately does not crash when the initial JWKS fetch fails; it starts with no keys and relies on refresh to recover, returning auth.jwks_unavailable in the meantime. With refresh dead, a brief IdP blip during gateway startup denies every request for that issuer indefinitely. A rolling restart during an IdP maintenance window is enough to trigger it. Comment: "the initial fetch failed and refresh has not yet recovered, the gateway didn't crash by design."

The failure is silent. Nothing errors, nothing logs. The task stops existing. The eventual symptom, sudden total denial for an issuer, has no visible connection to a startup-time cause.

PPE's own tests cannot catch it. They run under #[tokio::test], where the runtime lives for the whole test, so the task survives. The bug exists only in the embedding pattern Praxis uses.

resolver.rs:360 is the only production tokio::spawn reachable from any initialize() in the workspace. The others are all inside test modules, so this is a single site.

Fix

The durable fix belongs in PPE, and the code already carries the signal it needs. UnknownKid and KeysUnavailable at the verify path are exactly "a refresh is needed." A timer is not required.

Preferred: refresh on demand, single-flighted. Trigger a re-fetch from the verify path when a token's kid is unknown or the store is empty, instead of on a ticker.

• No background task, so no runtime assumption, so no way for an embedder to break it invisibly
• Works under any embedding, including the WASM direction
• More responsive than the ticker it replaces. A one hour interval means up to an hour of denial after a rotation; on demand recovers on the first token that needs it
• Testable without timing games

Hard requirement: rate limit per issuer. Unknown-kid is attacker triggerable with an unauthenticated request, so without a minimum interval between fetches this is a JWKS amplification DoS against your own IdP. Single-flight plus a floor on fetch frequency, both mandatory, not optional hardening.

Alternative: hand the task to the host. Return the refresh future for the host to spawn on its serving runtime, matching the injected-transport shape. Architecturally consistent, but it keeps the timer, keeps the runtime dependency, and adds public API surface. Take this only if on-demand refresh proves infeasible.

Praxis-side mitigation, independent of the above. Store the init Runtime in PolicyFilter rather than letting it drop. Restores intended behavior today with no PPE change and no API break. It costs a permanent thread and quietly becomes the home for anything else that spawns during init, so treat it as a stopgap rather than the destination.

A third reading is available: declare "the runtime calling initialize() must outlive the engine" as PPE's contract, making this purely a Praxis bug. Legitimate, but it is an invisible contract that is trivially violated, and it blocks the sync and WASM work. Not recommended as the only response.

Acceptance Criteria

• JWKS keys refresh without depending on a background task surviving the runtime that called initialize()
• A token with an unknown kid triggers at most one in-flight refresh per issuer
• A minimum interval between refresh attempts per issuer, so unauthenticated traffic cannot drive unbounded fetches at the IdP
• Recovery from a failed initial fetch, so auth.jwks_unavailable is transient rather than terminal
• Recovery from key rotation, so auth.unknown_kid resolves on the next fetch rather than at restart
• A regression test that drives initialize() on a runtime which is then dropped, and asserts refresh still works. This is the shape the current tests miss
• No production tokio::spawn reachable from initialize()
• The docstring at config.rs:252 matches actual behaviour
• If any runtime lifetime requirement remains, it is stated in the Plugin::initialize docs rather than implied

Notes

Praxis-side mitigation and the PPE fix are not mutually exclusive. Shipping the mitigation first is reasonable given the boot failure mode.