Ignore soft-404 pages when scraping product prices

[Alien10140]

Summary

• Detect common soft-404 markers (title text, Shopify template-404, canonical /404, etc.)
• Clear any scraped price and set availability to https://schema.org/Discontinued

Why

Soft 404 HTML still has a title and often a promotional banner with a dollar amount (e.g. “Orders Over $899!”). Naive price selectors then store that banner amount as the product price, which shows up as a false “deal.”

Related discussion: out-of-stock / unavailable URL handling (#166).

Test plan

• Unit tests for soft-404 detection and scrape behavior (banner price dropped)
• Manually: scrape a known soft-404 product URL with a banner dollar amount — price should be empty / discontinued
• CI green

Made with Cursor

Summary by CodeRabbit

• Bug Fixes

  • Improved soft-404 detection by analyzing scraped page titles and HTML content for common not-found/discontinued markers.
  • For pages identified as unavailable, pricing values are no longer stored.
  • Marked such pages with https://schema.org/Discontinued to prevent misleading availability/threshold-based pricing.

• Tests

  • Added unit coverage for soft-404 behavior, including “banner/threshold” scenarios.
  • Added detection tests for multiple 404-like markers and non-matching content.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4385536c-e811-4d7a-864b-7b0869850ecd

📥 Commits

Reviewing files that changed from the base of the PR and between 13f700a and 648b4ac.

📒 Files selected for processing (1)

• tests/Unit/Services/ScrapeUrlTest.php

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/Unit/Services/ScrapeUrlTest.php

---

📝 Walkthrough

Walkthrough

Changes

Soft 404 scraping guards

Layer / File(s)	Summary
Not-found detection and result guarding app/Services/ScrapeUrl.php	Scraped titles and bodies are checked for not-found markers; matching results have price cleared and availability set to https://schema.org/Discontinued.
Soft-404 behavior coverage tests/Unit/Services/ScrapeUrlTest.php	Tests verify threshold prices are suppressed and common not-found markers are detected without misclassifying a product page.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: ignoring soft-404 pages during product price scraping.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/Unit/Services/ScrapeUrlTest.php (1)

280-290: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover canonical /404 detection independently.

Line 280 omits the canonical branch; the integration fixture already matches via its title and template-404 class. Add direct cases for both rel/href attribute orders so a canonical-regex regression is caught.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/Unit/Services/ScrapeUrlTest.php` around lines 280 - 290, Add direct
assertions to test_looks_like_not_found_page_detects_common_markers for
canonical /404 links, covering both rel-before-href and href-before-rel
attribute orders. Keep the existing marker and non-404 assertions unchanged so
canonical detection is tested independently of title or template-404 matching.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/Unit/Services/ScrapeUrlTest.php`:
- Around line 280-290: Add direct assertions to
test_looks_like_not_found_page_detects_common_markers for canonical /404 links,
covering both rel-before-href and href-before-rel attribute orders. Keep the
existing marker and non-404 assertions unchanged so canonical detection is
tested independently of title or template-404 matching.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 202d840a-f6c3-4956-b34b-452147e2a5b0

📥 Commits

Reviewing files that changed from the base of the PR and between ed901c7 and 13f700a.

📒 Files selected for processing (2)

• app/Services/ScrapeUrl.php
• tests/Unit/Services/ScrapeUrlTest.php

[Alien10140]

Addressed CodeRabbit nit: added direct canonical /404 assertions for both rel/href attribute orders.

[jez500]

thanks, merged here #214 as it needed a bunch of fixes