BUY-61977: stop caching degraded envelopes in search + deals - #216
Open
BuyWhere wants to merge 1198 commits into
Open
BUY-61977: stop caching degraded envelopes in search + deals#216BuyWhere wants to merge 1198 commits into
BuyWhere wants to merge 1198 commits into
Conversation
…56497 round 2) fix(seo): optimize meta titles for remaining 2 zero-click pages (BUY-56497 round 2)
Remove 64 hardcoded /best-* entries from STATIC_SITEMAP_ROUTES that were also auto-derived from the seoLandingPages registry, causing every /best-* slug to appear twice in sitemap-pages.xml (211 total, 147 unique). The registry auto-derivation at Object.keys(seoLandingPages) already covers all 147 slugs. Removing the duplicates drops sitemap-pages.xml from 211 to 147 /best-* loc entries, eliminating crawl budget waste. Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Cron wrapper: scripts/run-buy-56579-disk-space-watchdog-cron.sh - Cron schedule: */5 * * * * (with .paperclip_env sourcing) - Thresholds: WARN <20GB, CRIT <5GB (creates Paperclip incident) - State-file dedup: 30 min critical, 60 min warn - Evidence: BUY-56579-evidence/apply-report.json - Verified: 57 GB free, exit 0, /dev/vda1 healthy
This reverts commit 3a8ad84.
- Removed untracked node_modules/ (root 752M + api 199M) - Git-rm 33 stale BUY-*/status/completion markdown files and fix scripts - Removed untracked BUY-*-evidence directories - Cleaned stale logs/, venv/, __pycache__, .bak files - Cleaned stale data/ runtime artifacts (pids, heartbeats, temp logs) - Updated .gitignore to prevent re-accumulation of transient artifacts
…s===3 (squashed) Root cause: releaseClientSafely checked client.state==='error' but pg's statement_timeout puts connections into PQTRANS_INERROR (transactionStatus===3) without changing client.state. Poisoned connections were returned to the pool via client.release(), contaminating subsequent queries with 'current transaction is aborted'. Fix: check client.transactionStatus === 3 instead of client.state === 'error'. All 4 DB client users (search_products, get_deals, list_categories, find_best_price) already use releaseClientSafely via origin/main. Co-authored-by: Dash <dash@buywhere.ai> Co-authored-by: Paperclip <noreply@paperclip.ing>
Add SEO landing pages that went missing: - /brands - brands listing hub - /brands/[slug] - brand deals pages - /brands/[slug]/deals - brand deals hub - /brands/[slug]/deals/earn - brand earn hub - /deals - deals page - /earn - earn page Co-Authored-By: Paperclip <noreply@paperclip.ing>
The /cheapest hub page was only listing 5 of 10 cheapest routes in its comparisonRows and fallbackProducts config, leaving these routes unreachable via internal navigation despite being indexed in the sitemap and returning HTTP 200: - /cheapest-dyson-us - /cheapest-ipad-us - /cheapest-ps5-us - /cheapest-samsung-tv-us - /cheapest-switch-us Adding them to the hub config restores internal link equity so the 5 previously orphaned pages stop losing PageRank and crawl budget. This is the re-applied fix for BUY-56822 after BUY-56629 closed without deploy verification.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reclaim ~400MB by removing stale ~40-day-old scraper outputs and transient evidence: - Git-rm 33 stale tracked data/<source>/products_*.jsonl files (May 14-16, ~427MB) - Remove 3 untracked carousell-sg products_20260516_*.jsonl files (~221MB) - Remove 43 untracked BUY-*-evidence directories (~760KB) - Clear __pycache__ directories (app/, scrapers/, scripts/) - Add reports/BUY-56824-workspace-disk-cleanup.md Preserves all source code modifications, live runtime state, and tracked work products (BUY-18309 shopify validation, BUY-56632 evidence, etc). Disk: 2.3GB -> 1.9GB, filesystem 65% -> 64%
…ent_timeout to 5s for find_best_price
… open.er-api Every 6h: re-fetch FX rates for currency conversions (USD/EUR/GBP/JPY/MYR/IDR/THB/PHP/VND/SGD) and upsert into fx_rates. - Primary source: frankfurter.app (ECB rates, free, keyless) - Fallback: open.er-api.com for currencies frankfurter doesn't carry (e.g. VND) - In-process scheduler via startFxRefreshScheduler() wired in index.ts - Admin endpoint POST /v1/admin/fx/refresh for manual triggers - In-memory cache in fxRatesLoader.ts (5-min TTL, EUR-based, USD-pivot) - buildProduct() and MCP findBestPrice use live rates from cache - fx_rates table migration in migrate.ts - npm scripts: fx-refresh / start:fx-refresh
- Initialize git repo to fix previous workspace_validation_failed error - Fix pre-existing Stripe apiVersion type error (2026-04-22 → 2026-05-27) - Verified fx_rates table exists (11 rows for EUR->USD/EUR/GBP/JPY/MYR/IDR/THB/PHP/VND/SGD + USD->USD) - Ran fxRefresh end-to-end: 11 rates upserted from [frankfurter, open.er-api] in 226ms - All rates freshly updated as of heartbeat timestamp
- Created canonical run-buy-57311-worker-wc-cycle-cleanup.sh covering ALL worker workspaces (not just Oracle), replacing 8 fragmented per-issue scripts. - Created idempotent setup-buy-57311-worker-node-disk-space-enforcement.sh using shared marker 'wc-cycle-cleanup-cron' that dedupes on re-install. - Added DEPRECATED notices to old scripts pointing to the new canonical one. - Consolidated crontab: removed 7 old cron entries (BUY-55411, BUY-55437, BUY-55448, BUY-56542, BUY-56941, BUY-57107, BUY-57166, BUY-57262), installed single entry running every 6 hours. - Ran initial cleanup pass: scanned 56 stale ndjson files, trashed all of them from Oracle workspace, disk at 63% (below 90% alert threshold).
Three-probe health check for the production MCP endpoint at api.buywhere.ai/mcp: 1. Server info (GET) - validates name=buywhere-catalog, protocol=mcp 2. tools/list (anonymous POST) - validates tool_count>0 and search_products present 3. tools/list (invalid auth POST) - informational, records auth_enforced flag Results saved to BUY-57227-evidence/ as timestamped JSON files.
…e 74 fragmented scripts - Added DEPRECATED by BUY-57327 markers to all 67 tracked and 7 fragmented per-issue worker wc-cycle-cleanup scripts, pointing to the canonical run-buy-57311-worker-wc-cycle-cleanup.sh - Added DEPRECATED markers to 3 old setup scripts and 2 untracked scripts that were missed by BUY-57311 consolidation - Added DEPRECATED marker to the overlapping disk-space-watchdog script - Verified crontab has exactly 1 consolidated WC cycle cleanup entry (0 */6 * * *) with wc-cycle-cleanup-cron marker - Ran canonical cleanup: 79 stale ndjson files trashed across 3 worker workspaces; disk at 64% (below 90% alert threshold)
…nup) - Added worker-node-disk-enforcement.sh — core enforcement engine that scans all workspaces, triggers wc-cycle-cleanup.sh on workspaces above 85% disk, and creates Paperclip critical incidents above 95% - Added run-buy-57336-worker-disk-enforcement.sh — cron wrapper with evidence reporting to BUY-57336-evidence/ - Added setup-buy-57336-worker-disk-enforcement.sh — idempotent cron installer (every 10 min, disk-enforcement-cron marker) - Installed cron: */10 * * * * — verified single entry - Initial pass: disk at 64% (71GB free), no workspaces above 85% threshold, no incidents triggered - Evidence: BUY-57336-evidence.md
Zero-byte cycle ndjson files cannot be meaningfully open at scale, and lsof is expensive on directories with 50K+ files. Skip the lsof check entirely for zero-byte files, returning "not open" immediately. This performance fix makes the cleanup pipeline invoked by the worker-node-disk-enforcement.sh engine more efficient.
…o-byte optimization
…ponse cache, and Redis ping timeout - Set 3s statement_timeout on each health endpoint DB query (runs, products, zombies) with automatic reset to 30s in finally() - Add 30s TTL in-memory response cache with configurable HEALTH_CACHE_TTL_MS env var - Add 1s race timeout on Redis ping to prevent blocking under connection pressure - Cached responses include cached: true field for observability - Internal monitoring header bypasses cache for real-time data Root cause: 3 sequential DB queries each using the global 30s statement_timeout, allowing worst case of 90s+ under PgBouncer pool exhaustion or concurrent load.
…nsafe schemes The expanded allowlist still missed some merchant domains (e.g. the product IDs in the QA repro), causing 403s. Root fix: destinationUrl is always resolved from our own DB (affiliate_links / products — admin-curated, not user input), so the guard now only blocks dangerous schemes (javascript:, data:, non-http). Any valid http(s) merchant URL from the catalog is permitted. Optional strict mode via AFFILIATE_STRICT_ALLOWLIST=1 re-enables exact-domain matching if ops ever need to lock down outbound redirects. Tests: 9 allowlist cases (incl. javascript:/data: blocked, arbitrary merchant allowed) + 4 existing BUY-60548 cases — all pass.
RECENT_SLICE_CAP=50000 was timing out at 8s against 1.5M fresh SG products (48h window). 2000 rows complete in <50ms on the products_sg_updated_at_idx index. The bounded CTE is a fallback - any results beat a degraded 8s timeout.
) Systematically replaced low-contrast text classes across product listing, comparison, and UI components: - text-slate-400 → text-slate-500/600/700 - text-gray-400 → text-gray-500/600 - text-indigo-200 → text-indigo-300/400 Key fixes: - SeoLandingPage: Product card merchant/category labels - ProductCard: Review counts, shipping info, strike-through prices - CompareProductsGrid: Empty states, comparison labels - USDealsSection: Deal timestamps, original prices - Category components: Filter counts, price indicators Targeted pages (previously showing 6-20 violations each): - Homepage, /laptop-singapore, /best-gaming-laptops-us, /search All critical product listing and comparison components now meet WCAG 2.1 AA 4.5:1 contrast requirements. Co-authored-by: Rex (codex_local) <rex@buywhere.ai>
PriceCard.tsx indexed a merchant dictionary by price.merchant, returning T | undefined for unknown merchants. TypeScript strict mode (next build type-check) failed with 'info is possibly undefined' at line 129, blocking all production deploys including the WCAG color-contrast fix. Adds a safe default fallback (slate palette + info icon) for unknown merchants. Co-authored-by: Rex (codex_local) <rex@buywhere.ai>
…207) getUSProducts/getSGProducts threw on non-OK API responses (e.g. 401 when the build-time API key is expired/invalid), crashing Next.js prerendering of /us/sitemap.xml and /api/us-sitemap.xml and blocking all production deploys. Wraps the API fetch in a catch that logs a warning and returns an empty product list, so sitemaps degrade gracefully to static-only routes instead of crashing the build. Co-authored-by: Rex (codex_local) <rex@buywhere.ai>
Footer copyright text (text-gray-400 on bg-gray-50 = ~2.5:1) and newsletter email placeholder both failed WCAG AA 4.5:1. Darkened to text-gray-500 (~4.6:1 on gray-50). Co-authored-by: Rex (codex_local) <rex@buywhere.ai>
The Starter 'Start Free Trial' and Pro 'Get Pro' CTAs on /pricing linked to dead anchor fragments (#starter-checkout, #pro-checkout). Point them at the existing /checkout route with the canonical plan slugs (pro, scale) so paid-tier clicks start a real Stripe checkout flow instead of going nowhere. Refs BUY-60006
…esponse - response.ts emits has_affiliate_tracking, is_affiliate, affiliate_disclosure - products.ts allows field filtering for the new disclosure fields - types/product.ts adds CanonicalProduct type fields - Tests cover precomputed affiliate URL + generated redirect disclosure Co-authored-by: Hex <7fb55262-e658-45e2-88c0-b0e8ccc5ad6c>
…esponse - response.ts emits has_affiliate_tracking, is_affiliate, affiliate_disclosure - products.ts allows field filtering for the new disclosure fields - types/product.ts adds CanonicalProduct type fields - Tests cover precomputed affiliate URL + generated redirect disclosure Co-authored-by: Hex <7fb55262-e658-45e2-88c0-b0e8ccc5ad6c>
…cbook-air-m3-12-countries-compared Symptom: 254 imp / 0 clk 28d at pos 9.8. Top queries are 100% 'cheapest country to buy macbook' framing, but meta title led with 'Price in 12 Countries' (mismatch). Fix: - title: 'MacBook Air M3 Price in 12 Countries (2026) — Cheapest is US99' → 'Cheapest Country to Buy MacBook Air M3 (2026) — From US99' - description: 'MacBook Air M3 price in 12 countries...' → 'Cheapest MacBook Air M3 in 2026 is in India at ₹94,900 (~$1,138); US at US$799 is the cheapest in the accessible global market...' - JSON-LD headline + description aligned to new title - H1 (# Cheapest MacBook Air M3 in 2026 — 12 Countries Compared) → # Cheapest Country to Buy MacBook Air M3 (2026) — From US99 Pattern: matches BUY-60521 (iPhone 16 vs 17) and wave-7 commit 347125c.
…icks allowed domains
Three contributing factors addressed: 1. Missing image host cdn.bestdenki.com.sg added to next.config remotePatterns 2. Data URI SVGs (buildCategoryImage fallback) now render with unoptimized prop on <Image> to avoid Next.js optimization pipeline failures during RSC 3. Fetch timeout reduced from 20s to 8s to prevent RSC timeouts on mobile edge networks (30-60s typical edge timeout) Plus: Added max-sm:overflow-visible to hero section for mobile rendering.
- Import HomeProductSearch component into the home page - Replace developer-focused hero with consumer-facing search experience - Add prominent search bar with country selector (US/SG) - Add consumer CTAs: Browse price comparisons, Developers: Get API access - Consumer heading: 'Find the best prices across every store.' - Keep developer audience content and SEO sections below the fold
…og/compare-headphones-singapore-2026 (#210) Symptom: 69 imp / 0 clk 28d at pos 8.3 (page 1 frontier). Top query is 'sony wh-1000xm5 price singapore 2026' but meta title led with category framing 'Best Wireless Headphones in Singapore 2026' — mismatch between search intent (specific product + price) and snippet. Fix: rewrite title/description/H1/JSON-LD headline to lead with the Sony WH-1000XM5 S$349 Shopee anchor, naming Lazada and Amazon alternatives to land the snippet. - title: 'Best Wireless Headphones in Singapore 2026 — Sony, Apple, Bose Price Comparison' (87 chars) -> 'Sony WH-1000XM5 Price Singapore (2026) — From S$349' (51 chars) - description: brand-list (200+ chars) -> S$349 Shopee anchor + 2 alternatives (109 chars) - H1 aligned to new title - JSON-LD Article headline/description aligned to new title - lastUpdatedAt + dateModified -> 2026-07-10 Pattern proven by BUY-61481 (cheapest-macbook-air-m3) and BUY-60521 (iphone-16-vs-17): same kind of rewrite produced 0 -> first clicks within T+14 window. Refs: BUY-61661
… by SEARCH_USE_TIER) Ports the RAM-fitting search_products tier + AND-first-then-OR + ts_rank ranking into the MCP handleSearchProducts keyword path, matching REST products.ts. Fixes MCP zero-result/latency (was querying the 254M archive ORDER BY updated_at, no relevance). Hybrid: any error falls through to the archive path. Flag-gated so deploy is behavior-neutral until SEARCH_USE_TIER=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014HMM7n7sjhWBYvKM8ttXcK
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014HMM7n7sjhWBYvKM8ttXcK
…rd zero-results) gin_fuzzy_search_limit=2000 (db+role level) was sampling GIN results, collapsing multi-word AND queries (MacBook Air, gaming laptop) to near-zero on both tier and archive. Fixed at DB level (ALTER DATABASE/ROLE) + belt-and-suspenders SET LOCAL in both tier query paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014HMM7n7sjhWBYvKM8ttXcK
…rns degraded=true
QA re-verified the dead-end on /search?q=laptop&country=sg — the upstream API now returns {products:[], total:0, degraded:true, hint:'...'} instead of a stale inflated-total payload. The previous catalogStale check (total > 0 && items empty) no longer fires. This commit reads the backend degraded flag, captures its hint, and renders an amber 'Catalog update in progress' status banner with retry + homepage CTAs. The original 'No products found' empty state is preserved for genuine zero-match queries (degraded=false).
…enAPI spec
- SearchResponse now returns { data: CanonicalProduct[], meta: SearchMeta }
- SearchMeta contains { total, limit, offset, response_time_ms, cached, degraded? }
- Updated all buildSearchResponse callers and inline degraded responses
- Fixes BUY-59842: API now matches OpenAPI documentation
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause of the bounded probe degraded responses: the deals handler caches its empty envelope for 1 hour whenever the read replica times out, and search tier + archive do the same on PG 57014 / 53200. Subsequent requests for the same cache key return cached degraded payloads even after the upstream recovers, which keeps the MCP fabricated-cache incident technically unresolved. - Tier search (`tryTierSearch`, line 154): skip cache write when response.meta.degraded - Archive search (line 1266): skip cache write when responseBody.meta.degraded - Deals (line 1479): skip cache write when the 57014/57000 path set degraded=true - Cache-key versions: `tier-default-v1` -> `tier-default-v1-buy-61977`, `deals:` -> `deals-buy-61977:`, invalidating any poisoned entries written before this guard landed. Existing cache-poison keys for deals (SG and US) will naturally fall out of the live API within an hour once the deploy lands; the versioned key bypasses them immediately.
…alth The auto-cleanup block for zombie ingestion runs used wrong column names (error instead of error_message, completed_at instead of finished_at), causing the UPDATE to silently fail. This meant zombie runs were never cleaned up by the health endpoint's internal monitoring path. Also update fix_ingestion_runs.py to read DATABASE_URL from env instead of hardcoded Docker host. Root cause of recurring zombie_runs > 0 on health checks. Co-Authored-By: Paperclip <noreply@paperclip.ing>
BuyWhere
force-pushed
the
buy-61977-degraded-cache
branch
from
July 13, 2026 10:30
7036ad1 to
ce15d73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The bounded production probe (BUY-61977) showed SG+US deals and US
/v1/products/search?q=laptopreturningmeta.degraded=true,total=0even after the auth fix in BUY-61976. Root cause: the deals handler caches its degraded envelope for 1 hour whenever the read replica times out (57014), and search tier + archive paths do the same. Subsequent requests return the cached degraded payload even after the upstream recovers, which keeps the MCPget_deals/search_productsmatrix marked unavailable.What
tryTierSearch(line 154): skip the 1h cache write whenresponse.meta.degraded === trueresponseBody.meta.degradeddegraded=truetier-default-v1→tier-default-v1-buy-61977,deals:→deals-buy-61977:so any keys written before this guard lands are bypassed immediatelytests/search.test.mjsto match the new cache-key prefixOut of scope (filed follow-ups)
search_productstier population — the search tier table (search_products) holds SG data only, so US keyword search falls through to the archive path and times out at 8s for broad queries. The smallest fix is to populate the tier for US; a different work item.Test
tscclean onproducts.ts(only pre-existingwebhooks.tsliteral-version error remains)tests/search.test.mjs: 22 pass / 10 fail on baseline (pre-existingBUY-61770shape-mismatch failures — they pass before this diff and fail at the same rate after it; verified by stashing all my edits and re-running the test against an untouchedproducts.ts).Deploy
Once approved, redeploy via Railway with an explicit
commitShaso theserviceInstanceDeployV2cache pitfall (per earlier BUYs) doesn't serve a stale checkout.