Replace Chromium with stealth versions

[kth8]

This project uses vanilla headless Chromium which sticks out like a neon sign for any kind of web scraping. There are many projects that modify Chromium to make it appear authentic to bypass rate limit/CAPTCHAs which can be used an easy drop in replacement, for example:

• https://github.com/arman-bd/chromiumfish
• https://github.com/CloakHQ/cloakbrowser
• https://github.com/clark-labs-inc/clark-browser

[MarcellM01]

Thanks for opening this and for the links.

This is definitely worth looking into, especially since browser fingerprinting seems to be one of the reasons crawling can behave differently inside Docker vs locally for DDG. I’ll investigate whether we can support a more realistic Chromium/browser backend or make the browser executable configurable, so users can plug in alternatives without hardcoding one specific fork.

I want to be a bit careful here though: the goal is not to turn TinySearch into an aggressive CAPTCHA/rate-limit bypass tool, but to make normal crawling more reliable and less fragile for legitimate agent/search use cases.

I’ll take this before the larger feature work, since crawler reliability is pretty core to the project.

[kth8]

Since this is already using Playwright, adding an option to connect over CDP instead of the built in Chromium instance will allow users the freedom to use whichever browser backend they want. This way you won't have to commit to any custom option. Also have the benefit of creating a lite variant of the docker.io/marcellm01/tinysearch image without an included Chromium browser to save space for those who want to go custom.

[MarcellM01]

Thanks that's a really good idea, so in that way if needed users can extend the functionality with custom browsers themselves! Also the idea of a lite variant is a good one since the introduction of SearXNG, the stack has become ever so heavier. Btw if you want to join the (admittedly very small and very early) community of TinySearch on discord feel free 😄 : https://discord.gg/3bEKgnnh8

[MarcellM01]

Sorry for the delay. I’m actively working on this feature now. I was just waiting for the system to mature a bit, but in its current state I think this will be a good addition to the URL scraping functionality. I’ll make sure that discovery of relevant links does not use this, since SearXNG is sufficient for that. However, if a custom external browser is required for scraping, it can be plugged in. Crawl4AI also natively supports this setup.