Skip to content

test: add keyless model listener filter demo#957

Open
mukeshbaphna wants to merge 1 commit into
katanemo:mainfrom
mukeshbaphna:mukesh/keyless-model-listener-filter-demo
Open

test: add keyless model listener filter demo#957
mukeshbaphna wants to merge 1 commit into
katanemo:mainfrom
mukeshbaphna:mukesh/keyless-model-listener-filter-demo

Conversation

@mukeshbaphna
Copy link
Copy Markdown
Contributor

@mukeshbaphna mukeshbaphna commented May 24, 2026

Summary

Adds a model-listener filter demo that exercises both input and output filters. The default path is fully local and requires no provider API keys or hosted model access. A second config lets developers run the same filter chain against the real OpenAI endpoint when they want provider-backed testing.

This follows up on #947 by proving the valid model-listener filter path end-to-end after invalid filter references now fail closed.

Changes

  • Add a fake OpenAI-compatible provider for deterministic local responses
  • Add config.openai.yaml for optional real OpenAI testing
  • Add an output redaction filter that removes sensitive markers from provider responses
  • Wire the model-listener demo to use both input_filters and output_filters
  • Make the compose setup select a config file via PLANO_CONFIG_FILE
  • Extend the smoke test to cover allowed input, blocked input, and output redaction
  • Add service-level pytest coverage for the local filter/provider services
  • Update README with the developer value and local test command

Why this helps developers

Developers can validate model-listener guardrails locally before using real providers. This catches filter wiring mistakes early, supports CI without secrets, and demonstrates request blocking plus response redaction without adding a passthrough agent layer.

Local Testing

uv run --with pytest --with fastapi --with httpx --with pydantic python -m pytest demos/filter_chains/model_listener_filter/test_services.py -q
# 4 passed
cd cli && uv run pytest test/test_config_generator.py -q
# 22 passed
bash -n demos/filter_chains/model_listener_filter/test.sh
git diff --check

Also validated the updated demo config with validate_and_render_schema.
Also validated config.openai.yaml with validate_and_render_schema.

Note: local pre-push npm run lint currently fails on unrelated existing frontend lint issues in packages/shared-styles and packages/ui; this PR only changes demos/filter_chains/model_listener_filter.

@mukeshbaphna mukeshbaphna force-pushed the mukesh/keyless-model-listener-filter-demo branch 4 times, most recently from c981880 to a535524 Compare May 24, 2026 08:25
@mukeshbaphna
Copy link
Copy Markdown
Contributor Author

Hi @Spherrrical @salmanap, this follow-up is ready for review when you have a chance. It builds on #947 by adding a local model-listener filter demo with both input and output filters, plus an optional OpenAI config for provider-backed testing.

@mukeshbaphna mukeshbaphna force-pushed the mukesh/keyless-model-listener-filter-demo branch 4 times, most recently from 7c56809 to 92e764a Compare May 24, 2026 14:30
run_test "Allowed request (joke)" 200 '{
run_test "Output filter redacts provider response" 200 '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Tell me a joke"}],
Copy link
Copy Markdown
Contributor Author

@mukeshbaphna mukeshbaphna May 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... this is not to reduce coverage. I wanted to make sure the smoke test covered all three aspects: allow, block, and redact.

Copy link
Copy Markdown
Contributor Author

@mukeshbaphna mukeshbaphna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment to explain why I removed additional "safe request: joke, expects 200"

@mukeshbaphna mukeshbaphna force-pushed the mukesh/keyless-model-listener-filter-demo branch from 92e764a to 3aadb2d Compare May 24, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant