URL transformations in ketch

[doctorjames]

I've noticed that when searches find Reddit links they're often actively blocked by Reddit even with headless chrome installed:

ketch search --scrape "ketch web search tool reddit"
warn: https://www.reddit.com/r/PiCodingAgent/comments/1sqk92y/what_websearch_webfetch_tool_are_you_using/ appears JS-rendered; configure browser for full content
---
url: https://www.reddit.com/r/PiCodingAgent/comments/1sqk92y/what_websearch_webfetch_tool_are_you_using/
title: Reddit - Please wait for verification
words: 0
---

However scraping old.reddit.com works fine:

Compare:

ketch scrape 'https://www.reddit.com/r/PiCodingAgent/comments/1sqk92y/what_websearch_webfetch_tool_are_you_using/'

and

ketch scrape 'https://old.reddit.com/r/PiCodingAgent/comments/1sqk92y/what_websearch_webfetch_tool_are_you_using/'

While you can add an instruction to the skill to always use the old URL, it's not able to affect search --scrape

Is it worth adding a configuration option to apply a regex replace on URLs before scraping? For news sites often fetching the RSS feed is cleaner than reading the main pages and so in those kinds of cases a manual transformation would be useful too.

For example:

https://www.theguardian.com/uk -> https://www.theguardian.com/uk/rss

(ketch's readability processing doesn't work on RSS, but the raw content still makes more sense to the LLM)

[1broseidon]

Shipped in v0.9.1: https://github.com/1broseidon/ketch/releases/tag/v0.9.1

Added url_rewrites — an ordered list of {match, replace} regex rules applied transparently before any fetch in scrape, search --scrape, and crawl. The original URL is preserved in output as url:; the rewritten URL appears as fetched_url: when different. Cache is keyed by the rewritten URL so aliases share one entry.

The two cases from your report:

# Reddit verification wall workaround
ketch config set url_rewrites '[
  {"match":"^https?://www\\.reddit\\.com/(.*)$","replace":"https://old.reddit.com/$1"}
]'

# Guardian /uk -> RSS feed
ketch config set url_rewrites '[
  {"match":"^(https://www\\.theguardian\\.com/uk)$","replace":"$1/rss"}
]'

Smoke-tested locally on your exact Reddit URL — frontmatter now shows both url: (original) and fetched_url: (old.reddit.com).

Does this solve the problem you described?

[1broseidon]

@doctorjames forgot to tag you on the reply

[doctorjames]

That's perfect! Thanks for the speedy implementation.