Skip to content

[Relay] feat(mcp): regression guard script for search_products (catches hardcoded fallback) - #15

Open
BuyWhere wants to merge 3 commits into
masterfrom
feature/mcp-regression-guard
Open

[Relay] feat(mcp): regression guard script for search_products (catches hardcoded fallback)#15
BuyWhere wants to merge 3 commits into
masterfrom
feature/mcp-regression-guard

Conversation

@BuyWhere

Copy link
Copy Markdown
Collaborator

What

Adds scripts/mcp_regression_guard.py - a small CLI that calls the deployed MCP /search_products with two distinct queries and asserts:

  1. The result sets differ between the two queries (catches 'same 3 products for every query' regressions).
  2. The first result of each query is NOT in a known-bad canned-fallback list (catches specific bad sets: ODS-AGM40E car batteries, Plastic Serving Set & Strainer, Ted Baker Gift Card, Soft Margaux 12 Bag, adidas Stan Smith, Crusher Tee).

Exit codes: 0 pass, 1 regression detected, 2 transport/parse error.

Why

The MCP endpoint at https://api.buywhere.ai/mcp has regressed to a hardcoded canned result set at least 4 times in 24h (BUY-55198 2026-06-22 00:19Z, BUY-55646 17:16Z, BUY-55752 20:56Z, plus follow-ups). Each regression means every MCP user gets wrong results. There is no deploy-gate that catches it. This script is the gate body - it can run as a CI step (preferred) or as a manual invocation.

Verification

Confirmed the guard correctly identifies the current regression by running it against https://api.buywhere.ai/mcp with the staging key:

MCP REGRESSION GUARD FAILED:
  - result set for laptop == result set for yoga mat (both = [Soft Margaux 12 Bag, Soft Margaux 10 Bag, Leon T-Shirt])
  - query laptop first result Soft Margaux 12 Bag matches known-bad canned fallback
  - query yoga mat first result Soft Margaux 12 Bag matches known-bad canned fallback

So the guard works on a broken endpoint. Once the upstream /v1/products is fixed, the guard will return exit 0 against healthy endpoints.

What this PR does NOT include

A GitHub Actions workflow file. Adding .github/workflows/mcp-regression-guard.yml requires a token with workflow scope, which this PR author does not currently have. Suggest the MCP deploy owner add the workflow file as a follow-up; the script here is self-contained and can be invoked from any CI environment.

Tracked in

  • BUY-55764 (child of BUY-55752) - Relay implementation ticket.

  • BUY-55752, BUY-55646, BUY-55198 - the regression instances this guard would have caught.

  • Relay

@BuyWhere

Copy link
Copy Markdown
Collaborator Author

👋 Bumping this PR — BuyWhere is a free, open MCP server for real-time product discovery across 100M+ live products (SG, SEA, US). We'd love to be included in your awesome list! Happy to answer any questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant