Effortless, Self-Healing, Stealth Web Scraping for the Modern Web.

---

💖 Sponsors

_{Want your logo here? Become a sponsor →}

---

---

⚡ What is Crawlingo?

Crawlingo is a next-generation, high-performance web scraping and crawler framework powered by a core compiled in Rust. It delivers self-healing selectors, stealth TLS/HTTP/2 fingerprinting, and SIMD-accelerated text anchors to make data extraction resilient, fast, and completely headless-browser-free.

Unlike scraper libraries that break when a website modifies a single CSS class, Crawlingo uses a localized DOM fingerprinting system with Jaro-Winkler similarity algorithms to automatically heal broken selectors in real-time.

Crawlingo ships with native, idiomatic SDKs for Python, Node.js (TypeScript), and Rust.

---

🚀 Key Features

• 🛡️ Self-Healing Selectors (Auto-Match): Automatically repairs broken CSS or XPath selectors in production using multi-dimensional DOM fingerprints stored in an embedded Sled database.
• 🌐 Stealth TLS/HTTP2 Profiles: Mimics genuine browser handshakes (JA3, HTTP/2 frames, user-agent headers) for Chrome, Firefox, and Safari to bypass Cloudflare, Akamai, and Turnstile without heavy headless browsers.
• ⚡ High-Throughput Rust Core: Achieves up to 3,500+ requests/second using Tokio's async I/O and Rayon's parallel CPU processing on commodity hardware.
• 💾 Zero-Copy FFI: Shared memory mappings ensure PyO3 (Python) and napi-rs (Node.js) bindings extract and parse datasets with near-zero overhead.
• 📊 Flexible Dataset Builder: Export extracted fields directly to structured JSON, CSV, or Parquet datasets in a single method call.
• 👁️ Reactive Watch Monitors: Polls websites on background intervals, automatically computing diffs and firing webhooks on content changes.

---

📊 Framework Comparison

Metric / Feature	Crawlingo	Scrapy	Playwright / Puppeteer
Language Bindings	Python · Node.js · Rust	Python	JS · Python · C# · Java
Throughput (50 concurrent)	~3,500 req/s	~500 req/s	~50 req/s
Memory Footprint (Idle)	2.4 MB	~50 MB	~200 MB
Self-Healing Selectors	✅ Built-in	❌ Manual Fix	❌ Manual Fix
Stealth TLS Fingerprinting	✅ Built-in	❌ Third-party	❌ Heavy plugins
Change Monitoring	✅ Built-in	❌	❌
CPU Acceleration	✅ SIMD	❌	❌

---

💻 Multi-Language Quick Start

🐍 Python SDK

pip install crawlingo

from crawlingo import Page, Session, Dataset

with Session() as s:
    s.auto_match(True).fetcher_tier("stealthy").rate_limit(5)

    result = (Dataset("https://shop.example.com", session=s)
              .field("title", "h1")
              .field("price", ".price", extraction_type="price")
              .field("in_stock", ".stock-badge")
              .build())

    print(result.to_dict())
    result.to_parquet("shop_data.parquet")

📘 Node.js (TypeScript) SDK

npm install crawlingo

import { Page, Session, Dataset } from 'crawlingo';

const session = new Session().autoMatch(true).fetcherTier("stealthy").rateLimit(5);

const result = await new Dataset("https://shop.example.com", session)
  .field("title", "h1")
  .field("price", ".price", { extractType: "price" })
  .field("in_stock", ".stock-badge")
  .build();

console.log(result.toDict());
result.toJsonFile("shop_data.json");

🦀 Rust SDK

[dependencies]
crawlingo = "1.0.0-alpha.1"
tokio = { version = "1", features = ["full"] }

use crawlingo::*;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let session = Arc::new(Session::new());
    session.set_auto_match(true);
    session.set_fetcher_tier("stealthy");

    let result = Dataset::new("https://shop.example.com", session)
        .with_field(DatasetField::new("title", "h1"))
        .with_field(DatasetField::new("price", ".price").with_extract_type(ExtractionType::Price))
        .build_async()
        .await?;

    println!("{:#?}", result.fields);
    Ok(())
}

---

⚙️ Advanced Configuration

Config Key / Env Var	Default	Description
fetcher_tier / CRAWLINGO_TIER	"standard"	"standard" (raw HTTP/2) or "stealthy" (browser fingerprint spoofing).
auto_match / CRAWLINGO_AUTO_MATCH	true	Enables Jaro-Winkler based self-healing for broken selectors.
rate_limit / CRAWLINGO_RATE_LIMIT	0 (Disabled)	Restricts request rate to N requests per second per host.
proxy_pool / CRAWLINGO_PROXIES	[]	List of proxy URLs to rotate in a round-robin pool.
fingerprint_db / CRAWLINGO_DB_PATH	".crawlingo"	File path to store DOM selector fingerprints in Sled DB.

---

📚 Resources & Community

Resource	Link
🌐 Official Website	crawlingo-docs.vercel.app
📖 Documentation	crawlingo-docs.vercel.app
🐍 PyPI Package	pypi.org/project/crawlingo
📦 npm Package	npmjs.com/package/crawlingo
🦀 crates.io	crates.io/crates/crawlingo
📋 Changelog	CHANGELOG.md
🤝 Contributing	CONTRIBUTING.md
🔐 Security Policy	SECURITY.md

---

📄 License

Crawlingo is open-source software licensed under the MIT License.