Investigate async `on_response_body` support

[liavweiss]

Description

on_response_body is currently synchronous in the HttpFilter trait
due to a Pingora constraint. External guardrail providers (and any
filter that needs async HTTP calls during response processing) require
an async version of this hook.

• Investigate the Pingora response_body_filter hook — it is defined as a sync fn, which propagates to Praxis's on_response_body
• Evaluate feasibility of a sync-to-async bridge at the Pingora boundary.
• Consider whether upstream Pingora changes are needed or if this can be solved within Praxis
• Document findings and recommended approach

Context:

• on_request_body is already async — this is only a response-side limitation
• "ai_guardrails: NeMo response-side guardrails" is blocked on this
• Any future filter that needs to make HTTP calls during response processing is also blocked

Motivation

Multiple features require async operations during response body
processing (AI guardrails response-side, HTTP callout filter for
responses, etc.). Without this, any filter that needs to call an
external service during response processing cannot be implemented.

Alternatives Considered

Running the external call synchronously with block_on — this risks
blocking the Pingora event loop and degrading proxy throughput. Needs
investigation to determine if it is safe in practice.

Area

Filter Pipeline

Parent Epic (optional)

#76

[liavweiss]

Investigation: Async on_response_body Support

Summary

on_response_body is synchronous because Pingora's
ProxyHttp::response_body_filter is a sync fn. This propagates
through the entire Praxis stack. The request-side equivalent
(request_body_filter) is async - the asymmetry is an intentional
Pingora design choice due to boxed-future allocation concerns.

The constraint originates in Pingora's ProxyHttp trait, not in
Praxis.

Options

Option A: Make response_body_filter async

Make response_body_filter an async fn - either in the
praxis-proxy/pingora fork ourselves, or by pulling an upstream
change if/when Cloudflare merges one.

Note: The Praxis-side work is the same regardless of who writes
the Pingora change. Even if upstream merges an async
response_body_filter, we still need to pull the new Pingora
version and update every Praxis filter that implements
on_response_body.

Upstream status:

Issue	Opened	Status
cloudflare/pingora#347	Aug 2024	Open - no merge after ~2 years
cloudflare/pingora#525	Feb 2025	Community PR - not merged
cloudflare/pingora#831	Mar 2026	Maintainer: "will likely make configurable". One of the contributors opened a small PR to fix, but no response yet

A community contributor (@gabioprisan) forked Pingora, made all
methods async, and reported 3x performance improvement for
I/O-bound workloads.

Files to change (Pingora fork):

• pingora-proxy/src/proxy_trait.rs - signature change
• pingora-proxy/src/proxy_h1.rs (~line 875) - add .await
• pingora-proxy/src/proxy_h2.rs (~line 820) - add .await
• pingora-proxy/src/lib.rs - downstream_response_task_filter
• pingora-core/src/modules/http/compression.rs - make impl async

Files to change (Praxis):

• protocol/src/http/pingora/handler/with_body.rs - async fn
• protocol/src/http/pingora/handler/response_body_filter.rs -
  async fn
• filter/src/pipeline/http.rs - execute_http_response_body ->
  async
• filter/src/filter.rs - on_response_body -> async fn
• All filters implementing on_response_body (token_count, A2A,
  response_store, compression, etc.)

Pros:

• Cleanest solution; matches request side exactly
• All filters gain async capability
• Proven by community fork (3x improvement for I/O workloads)

Cons:

• Large coordinated change across fork + Praxis
• Further diverges from upstream Pingora (if we do it ourselves)
• Breaking change for external HttpFilter implementors
• Boxed future allocation cost per chunk (this is unavoidable -
  any solution that enables async in on_response_body pays this
  cost, whether via Option A or Option B)

---

Option B: block_in_place bridge (immediate unblock)

Use tokio::task::block_in_place + Handle::block_on inside the
guardrails filter's sync on_response_body to run async code
inline.

Existing precedent in Praxis:

This is already used in the response store filter:

// filter/src/builtins/http/ai/openai/responses/store/filter.rs
let handle = tokio::runtime::Handle::current();
tokio::task::block_in_place(|| {
    handle.block_on(store.upsert_response(record))
})

Pros:

• No Pingora fork changes needed
• Small, localized change (only in ai_guardrails)
• Proven pattern already in the Praxis codebase
• Response IS modified before reaching the client - because
  block_in_place blocks inline, the filter can modify or reject
  the response body before it reaches the client

Why not the extensions.md alternative? The docs suggest
"spawn a background task and communicate via a channel" for
async I/O in response processing. This does not work for
guardrails because the response is already sent to the client
by the time the background task completes - you cannot
retroactively block or redact content that has already been
delivered. The background-task approach only works for
fire-and-forget work (e.g. logging, storing records).

Cons:

• Blocks a Tokio worker thread for the NeMo HTTP call duration
  (~1-5s)
• Under high concurrency, can exhaust worker threads
• Contradicts documented guidance: "Do not block the thread with
  block_on or heavy computation"
  (docs/filters/extensions.md)
• Cannot cancel if client disconnects

---

Option C: Wait for upstream Pingora

Do nothing; monitor
cloudflare/pingora#831.
Maintainer said "will likely make configurable."

Pros:

• Zero effort on our side for the Pingora change itself

Cons:

• 2 years with no merge; no timeline commitment
• Cloudflare has no internal need for this feature
• Blocks our response-side guardrails indefinitely
• When it does land, we still need to do all the Praxis-side work
  (same as Option A)

Recommendation

I would suggest starting with Option B (block_in_place) to
unblock response-side guardrails. It's a proven pattern in the
codebase and lets us validate the full request+response flow
end-to-end without touching the Pingora fork.

That said, any future external provider that evaluates response
bodies (e.g. Bedrock guardrails) will hit the same sync constraint
and need the same workaround. If we end up with multiple providers
all working around this limitation, it may be worth investing in
Option A - making on_response_body async in the fork. It's
more effort and riskier since it touches every filter that
implements on_response_body, but it solves the problem once for
all current and future filters.

[shaneutt]

Thank you for taking the initiative here. It may come down to us needing to improve our fork, and therefore diverging significantly and potentially ending up in the unfavorable "hard fork" situation, but I would like to try and avoid that.

Marking this as holding-pattern for now, as we'll need further discussion and investigations to determine the best course.

[liavweiss]

Thank you, Shane.

When we get to response-side guardrails, we'll use block_in_place as a scoped workaround. That keeps it contained to the ai_guardrails filter without touching the fork.

We can revisit the fork question later (depends on what Pingora's maintainers will decide to do).