feat: add llms.txt documentation index source

README.md

@@ -5,6 +5,7 @@ AI-friendly docs/spec snapshot scraper for:
 - normal websites (`web`)
 - explicit URL sets (`url-list`)
 - public GitHub documentation trees (`github-tree`)
+- documentation indexes exposed via `llms.txt` (`llms-txt`)
 
 It creates **versioned snapshots**, a **latest** mirror, **per-page metadata headers**, **URL inventories**, and **change summaries** so an agent can work against a reproducible local corpus instead of scraping ad hoc on every prompt.
 
@@ -39,9 +40,12 @@ This tool exists to fill that gap.
   - unchanged URLs
 - **Automatic folder organization** by source name
 - **Website crawling** with host restriction and include/exclude regex filters
+- **`llms.txt` index ingestion** for docs sites that publish one
 - **GitHub tree ingestion** for public repos using the GitHub tree API plus raw file fetches
 - **Explicit URL-list mode** for targeted snapshots
 - **URL inventory files** in both JSON and TXT formats
+- **Binary paper assets** for direct PDF URLs and discovered arXiv/PDF links
+- **Asset manifests** with byte length, content type, canonical URL, and SHA-256 for idempotent verification
 
 ## Install
 
@@ -65,6 +69,8 @@ spec-snapshot-scraper run --config examples/ircv3-web.json
 
 Every run writes to the configured `outputDir`:
 
+> `outputDir` is resolved **relative to the config file location**, so example configs inside `examples/` can intentionally target `../output/...` in the repository root.
+
 ```text
 outputDir/
   latest/
@@ -137,6 +143,28 @@ Use this when you want a small curated set of exact URLs.
 
 This is useful when you do not want the crawler to discover links.
 
+Direct PDF URLs are downloaded as binary assets under `assets/` and receive a companion Markdown stub under `pages/` so RAG pipelines can index metadata without corrupting the original file. Assets are idempotent across runs: when a previous manifest entry and local file still match by SHA-256, the scraper reuses the verified local bytes instead of refetching the paper. To discover linked papers from HTML or Markdown pages, enable `capturePaperAssets`:
+
+```json
+{
+  "outputDir": "./output/research-papers",
+  "sources": [
+    {
+      "name": "research-papers",
+      "type": "url-list",
+      "capturePaperAssets": true,
+      "allowPaperHosts": ["arxiv.org", "ru.iis.sociales.unam.mx"],
+      "urls": [
+        "https://arxiv.org/abs/2103.00104",
+        "https://ru.iis.sociales.unam.mx/bitstream/IIS/5684/2/sociosemiotica_y_cultura.pdf"
+      ]
+    }
+  ]
+}
+```
+
+arXiv abstract URLs such as `https://arxiv.org/abs/2103.00104` are normalized to their full PDF asset URL, `https://arxiv.org/pdf/2103.00104`.
+
 ### 3) `github-tree`
 
 Use this for public GitHub documentation repos.
@@ -145,6 +173,8 @@ The tool fetches the public tree via GitHub's tree API and then downloads matchi
 
 This is better than manually maintaining large raw URL lists when the upstream docs repo changes often.
 
+For paper-index repositories, `capturePaperAssets` can also be enabled on a `github-tree` source. The scraper will parse fetched Markdown, discover direct PDF links and arXiv abstract links, download the paper assets, and write `_assets.json` next to `_manifest.json`.
+
 Example:
 
 ```json
@@ -165,6 +195,31 @@ Example:
 }
 ```
 
+### 4) `llms-txt`
+
+Use this when the upstream docs site publishes a machine-readable documentation index at `llms.txt`.
+
+This is ideal for projects like Iroh where the docs explicitly recommend discovering the full page set from that file first.
+
+Example:
+
+```json
+{
+  "outputDir": "./output/iroh-llms",
+  "sources": [
+    {
+      "name": "iroh-llms",
+      "type": "llms-txt",
+      "llmsUrl": "https://docs.iroh.computer/llms.txt",
+      "allowHosts": ["docs.iroh.computer"],
+      "includeUrlPatterns": [
+        "^https://docs\\.iroh\\.computer(?:/.*)?$"
+      ]
+    }
+  ]
+}
+```
+
 ## Updating when upstream changes
 
 Just rerun the same config:
@@ -182,6 +237,13 @@ The tool will:
 
 That gives you a reproducible audit trail for spec drift.
 
+## Ready-made configs
+
+- `examples/iroh-llms.json` — snapshots docs discovered from `https://docs.iroh.computer/llms.txt`
+- `examples/rust-stable-web.json` — crawls the stable Rust docs surface (`book`, `reference`, `rustdoc`, `cargo`, `std`) with a bounded page budget
+- `examples/docsrs-about.json` — snapshots the stable docs.rs about/build/metadata pages without touching arbitrary crate docs
+- `examples/quinn-docs.json` — snapshots the key Quinn docs.rs pages plus the Quinn guide for QUIC transport tuning work
+
 ## How metadata is stored for AI consumption
 
 Every stored page begins with a YAML-style metadata header, for example:

examples/docsrs-about.json

@@ -0,0 +1,22 @@
+{
+  "outputDir": "../output/docsrs-about",
+  "sources": [
+    {
+      "name": "docsrs-about",
+      "type": "url-list",
+      "urls": [
+        "https://docs.rs/about",
+        "https://docs.rs/about/badges",
+        "https://docs.rs/about/builds",
+        "https://docs.rs/about/metadata",
+        "https://docs.rs/about/redirections",
+        "https://docs.rs/about/download",
+        "https://docs.rs/about/rustdoc-json",
+        "https://docs.rs/about/releases",
+        "https://docs.rs/about/build-queue",
+        "https://docs.rs/about/shorthand-urls",
+        "https://docs.rs/about/privacy"
+      ]
+    }
+  ]
+}

examples/iroh-llms.json

@@ -0,0 +1,17 @@
+{
+  "outputDir": "../output/iroh-llms",
+  "sources": [
+    {
+      "name": "iroh-llms",
+      "type": "llms-txt",
+      "llmsUrl": "https://docs.iroh.computer/llms.txt",
+      "allowHosts": ["docs.iroh.computer"],
+      "includeUrlPatterns": [
+        "^https://docs\\.iroh\\.computer(?:/.*)?$"
+      ],
+      "excludeUrlPatterns": [
+        "^https://docs\\.iroh\\.computer/llms\\.txt$"
+      ]
+    }
+  ]
+}

examples/pdf-assets.json

@@ -0,0 +1,15 @@
+{
+  "outputDir": "./output/pdf-assets",
+  "sources": [
+    {
+      "name": "paper-assets",
+      "type": "url-list",
+      "capturePaperAssets": true,
+      "allowPaperHosts": ["arxiv.org", "ru.iis.sociales.unam.mx"],
+      "urls": [
+        "https://arxiv.org/abs/2103.00104",
+        "https://ru.iis.sociales.unam.mx/bitstream/IIS/5684/2/sociosemiotica_y_cultura.pdf"
+      ]
+    }
+  ]
+}

examples/quinn-docs.json

@@ -0,0 +1,20 @@
+{
+  "outputDir": "../output/quinn-docs",
+  "sources": [
+    {
+      "name": "quinn-docs",
+      "type": "url-list",
+      "urls": [
+        "https://docs.rs/quinn/latest/quinn/",
+        "https://docs.rs/quinn/latest/quinn/struct.TransportConfig.html",
+        "https://docs.rs/quinn/latest/quinn/struct.Connection.html",
+        "https://docs.rs/quinn/latest/quinn/struct.Endpoint.html",
+        "https://docs.rs/quinn/latest/quinn/struct.MtuDiscoveryConfig.html",
+        "https://docs.rs/quinn/latest/quinn/struct.IdleTimeout.html",
+        "https://quinn-rs.github.io/quinn/",
+        "https://quinn-rs.github.io/quinn/quic.html",
+        "https://quinn-rs.github.io/quinn/getting-started.html"
+      ]
+    }
+  ]
+}

examples/rust-stable-web.json

@@ -0,0 +1,25 @@
+{
+  "outputDir": "../output/rust-stable-web",
+  "sources": [
+    {
+      "name": "rust-stable-web",
+      "type": "web",
+      "rootUrl": "https://doc.rust-lang.org/stable/",
+      "seedUrls": [
+        "https://doc.rust-lang.org/stable/book/",
+        "https://doc.rust-lang.org/stable/reference/",
+        "https://doc.rust-lang.org/stable/rustdoc/",
+        "https://doc.rust-lang.org/stable/cargo/",
+        "https://doc.rust-lang.org/stable/std/"
+      ],
+      "allowHosts": ["doc.rust-lang.org"],
+      "includeUrlPatterns": [
+        "^https://doc\\.rust-lang\\.org/stable/(?:book(?:/.*)?|reference(?:/.*)?|rustdoc(?:/.*)?|cargo(?:/.*)?|std(?:/.*)?)$"
+      ],
+      "excludeUrlPatterns": [
+        "^https://doc\\.rust-lang\\.org/stable/std/(?:arch|primitive|keyword|core|alloc)(?:/.*)?$"
+      ],
+      "maxPages": 400
+    }
+  ]
+}

package.json

@@ -7,8 +7,10 @@
     "spec-snapshot-scraper": "src/cli.mjs"
   },
   "scripts": {
+    "test": "node --test tests/*.test.mjs",
     "run:ircv3": "node src/cli.mjs run --config examples/ircv3-web.json",
-    "run:ircv3-github": "node src/cli.mjs run --config examples/ircv3-github-tree.json"
+    "run:ircv3-github": "node src/cli.mjs run --config examples/ircv3-github-tree.json",
+    "run:iroh": "node src/cli.mjs run --config examples/iroh-llms.json"
   },
   "keywords": [
     "scraper",