Transform any documentation site into perfectly structured Markdown — optimised context for your coding agents and LLMs.
- Recursive web crawler — crawls a documentation site up to a configurable depth (1–5 hops) and page limit (up to 200 pages), staying within the same origin and base path.
- Cloudflare / bot-protection bypass — attempts a realistic
fetchfirst; automatically falls back to a headless Chromium browser (viapuppeteer-core) when a Cloudflare challenge is detected. - Smart HTML → Markdown conversion — strips nav, footer, sidebar, ads, and scripts with
cheerio, then converts the clean content to fenced-code, ATX-heading Markdown viaturndown. - Real-time progress stream — scraping progress is streamed to the UI over Server-Sent Events so you see every page as it is processed.
- Compiled output — results are assembled into a single Markdown document with a table of contents, page anchors, and source URLs.
- Multiple export formats — download the scraped content as
.md,.txt,.json, or.html. - Copy to clipboard — one-click copy of the full compiled Markdown.
- Raw / Rendered preview — toggle between raw Markdown and a rendered
react-markdownpreview inside the app. - Scrape history — the last 20 scrape sessions are persisted to
localStorageso you can restore previous results instantly. - AI Summarize & Q&A — powered by OpenRouter; choose from Gemini 2.0 Flash, Gemini 2.5 Pro, Claude 3.5 Sonnet, GPT-4o, or GPT-4o Mini to summarise the docs or ask questions about them. Responses are streamed in real time.
- Advanced crawl settings — configurable page limit (10 / 25 / 50 / 100) and crawl depth via an expandable panel.
- Concurrent scraping — up to 5 pages are fetched in parallel per batch for fast coverage.
| Layer | Technology |
|---|---|
| Framework | Next.js 15 (App Router) |
| Language | TypeScript 5 |
| Styling | Tailwind CSS 4 |
| HTML parsing | cheerio |
| Markdown conversion | turndown |
| Headless browser | puppeteer-core |
| AI gateway | OpenRouter |
| Animations | Motion |
| Icons | Lucide React |
- Node.js 18+
- A Chromium-based browser installed locally (for Cloudflare fallback during development)
- An OpenRouter API key (optional — only needed for the AI features)
git clone https://github.com/musamusakannike/docscraper-ai.git
cd docscraper-ai
npm installCopy the example file and fill in your values:
cp .env.example .env.local| Variable | Required | Description |
|---|---|---|
BROWSER_WS_ENDPOINT |
Optional | WebSocket endpoint for a remote headless browser service (e.g. Browserless, BrowserBase). Falls back to local Chrome when not set. |
Note: The OpenRouter API key is entered directly in the UI and stored in
localStorage. It is never sent to any server other than OpenRouter.
npm run devOpen http://localhost:3000 in your browser.
npm run build
npm start- Paste the root URL of any documentation site (e.g.
https://docs.example.com). - (Optional) Expand Advanced Settings to adjust the page limit and crawl depth.
- Click Optimize Context and watch pages being scraped in real time.
- Once complete, copy, preview, or export the compiled Markdown.
- (Optional) Open the AI Summarize & Q&A panel, enter your OpenRouter API key, pick a model, and either summarise the docs or ask specific questions about them.
For production deployments, set BROWSER_WS_ENDPOINT to a Puppeteer-compatible WebSocket endpoint so the server can render JavaScript-heavy or Cloudflare-protected pages:
- Browserless —
wss://chrome.browserless.io?token=YOUR_TOKEN - BrowserBase
- Any self-hosted
puppeteer-core-compatible endpoint
When BROWSER_WS_ENDPOINT is not set, the scraper uses the local Chrome installation at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome.
Contributions, issues, and feature requests are welcome. Feel free to open an issue or submit a pull request.
- Fork the repository
- Create your feature branch (
git checkout -b feat/my-feature) - Commit your changes (
git commit -m 'feat: add my feature') - Push to the branch (
git push origin feat/my-feature) - Open a Pull Request
Distributed under the MIT License.