diff --git a/feishu-doc-scraper/SKILL.md b/feishu-doc-scraper/SKILL.md index 41b730cd..76853bc4 100644 --- a/feishu-doc-scraper/SKILL.md +++ b/feishu-doc-scraper/SKILL.md @@ -1,7 +1,7 @@ --- name: feishu-doc-scraper description: Extract Feishu (Lark) Docs, Wiki pages/collections, spreadsheets, and Minutes (妙记) transcripts into faithful local Markdown via the lark-cli API (no LLM rewriting of the body; browser-DOM fallback when lark-cli can't reach the content). Use whenever the source is a Feishu/Lark URL and fidelity matters — 导出飞书文档/合集/妙记转写, 把飞书 wiki/知识库转 markdown, archiving a Feishu collection, exporting a 妙记 transcript, or saving a Feishu page — even if the user only says clipping, archiving, converting, or "save this". Also covers the owner-exported .docx → faithful Markdown path. -compatibility: Primary path needs the `lark-cli` binary (npm `@larksuite/cli`, verified 1.0.32, 2026-05) authenticated to the target tenant. Fallback path needs a browser automation surface with an authenticated session (Chrome DevTools MCP / Browser Use / Computer Use). docx path needs `python-docx` and a docx→md converter (the bundled doc-to-markdown skill or pandoc). +compatibility: Primary path needs the `lark-cli` binary (npm `@larksuite/cli`; verified 1.0.32, 2026-05, and re-verified 1.0.80, 2026-08 — the `.data.markdown` field is null on 1.0.80 and the pandoc/`source.html` path in step 3 is load-bearing there) authenticated to the target tenant. Fallback path needs a browser automation surface with an authenticated session (Chrome DevTools MCP / Browser Use / Computer Use). docx path needs `python-docx` and a docx→md converter (the bundled doc-to-markdown skill or pandoc). argument-hint: "[feishu-url-or-output-path]" --- @@ -27,6 +27,8 @@ Is the source a Feishu/Lark URL (wiki / docx / sheets / minutes / base)? └── you were handed an exported .docx (not a URL) → PATH B ``` +⚠️ **`base` (Bitable) is not actually operationalized in Path A yet** — the extractor only records the token (`DISPATCH["url-base"]`: *"Bitable API (outside this skill) — record token"*) and Step 2 below has no row for it. Listed here for completeness of what a Feishu URL can be, not as a claim that Path A extracts Bitable content end-to-end. + A collection/hub is just a docx whose body references other docs — **Path A handles it by recursively following the reference graph**, not by visiting pages in a browser. ## Path A — lark-cli API extraction (primary) @@ -58,30 +60,53 @@ This does not conflict with any "Claude/Anthropic domains must use the proxy" ru lark-cli docs +fetch --doc --format json > /tmp/fetch.json 2> /tmp/fetch.err # ≤1.0.32: clean Markdown in .data.markdown. # 1.0.55: body moved to .data.document.content as HTML (.data.markdown is null). -if jq -e -r '.data.markdown // empty' /tmp/fetch.json > source.md && [ -s source.md ]; then +if jq -e -r '.data.markdown // empty' /tmp/fetch.json > ".md" && [ -s ".md" ]; then : # got clean Markdown directly else - jq -r '.data.document.content' /tmp/fetch.json | pandoc -f html -t gfm > source.md + jq -r '.data.document.content' /tmp/fetch.json > ".html" + pandoc -f html -t gfm ".html" > ".md" fi ``` -`--format markdown` is **not** a valid value (lark-cli warns and falls back to json). Keep stdout and stderr separate — a harmless `[deprecated]` line goes to stderr, and piping `2>/dev/null` *and* `jq` together produced a false `Exit code 5` in practice. The body must reach disk via `jq`/`pandoc`, never retyped or summarized by the model — paraphrasing silently corrupts source text, the single most important fidelity rule. (pandoc only re-renders HTML structure to Markdown; it does not rewrite prose, so fidelity holds.) +⚠️ **`` must be a distinct name per document, never the literal string "source" reused across fetches.** Step 4 below fetches multiple documents (the hub, then every child it references) into the same working directory — running this exact snippet twice with a hardcoded `source.md`/`source.html` would let the second fetch silently overwrite the first document's saved body before you ever got to check it. The rest of this section still says "`source.html`"/"`source.md`" as shorthand for *whichever document's own saved files you're currently looking at* — not one shared filename for the whole collection. + +⚠️ **On this pandoc branch, `pandoc -f html -t gfm` silently strips several of Feishu's custom embedded tags — verified against real documents, 2026-08-16/17, and the damage is worse the deeper you look:** +- **whiteboard** (``, an inline diagram block) vanishes with zero trace. Confirmed on a real document: 3 raw tags in `.data.document.content` → 0 trace in the pandoc-converted `source.md` (`grep -c whiteboard` found 3 hits in the raw HTML, 0 in the converted Markdown). +- **mention-doc** — whose real raw tag is ``, not the `Title` shape this skill originally assumed — also vanishes with zero trace, and worse than just losing the token/type: the title lives in a `title="…"` attribute and the tag body is empty, so pandoc drops the whole element and not even the bare title text survives. +- **sheet** is unconfirmed either way (no real sheet-reference document was found to test) — treat it as capable of silently vanishing too until proven otherwise. +- **image** — whose real raw tag is a standard `…`, not `` — does *not* vanish: pandoc passes the raw `` element through mostly intact (`src`/`id`/`href`/`width`/`height`/`alt` survive; `name=` is dropped; `mime=`/`scale=` are renamed `data-mime=`/`data-scale=`). +- **lark-table** turns out not to be a real tag for ordinary docx tables at all — they use plain `` HTML, and pandoc generally converts them intact (clean GFM pipe-table syntax for single-paragraph cells, or a raw `
` HTML block for multi-paragraph cells) — not part of this silent-loss class. + +Because the loss is real and type-dependent, **extraction (step 4) and the residual-tag check (step 5) must operate on `source.html`, never on `source.md`, whenever this pandoc branch was taken.** On the `.data.markdown` branch (≤1.0.32), if it is still reachable at all, the tags already survive as literal text directly in `source.md`, and checking `source.md` there remains correct — this caveat is specific to the pandoc fallback, which is the **current default**: `.data.markdown` was `null` in every real document checked (11/11 — 3/3 fresh fetches on the currently-installed lark-cli 1.0.80, plus 8/8 archived 2026-07-25 fetches), so whether the old branch is still reachable on any current lark-cli build was not confirmed. + +`--format markdown` is **not** a valid value (lark-cli warns and falls back to json). Keep stdout and stderr separate — a harmless `[deprecated]` line goes to stderr, and piping `2>/dev/null` *and* `jq` together produced a false `Exit code 5` in practice. The body must reach disk via `jq`/`pandoc`, never retyped or summarized by the model — paraphrasing silently corrupts source text, the single most important fidelity rule. (pandoc only re-renders HTML structure to Markdown; it does not rewrite prose — the tag-stripping above is a structural loss, not a prose-fidelity one, which is why source.html must stay on disk and stay authoritative for rich-media references.) + +**4. If it's a collection/hub, follow the reference graph (BFS).** The hub body contains `` (real raw tag: ``), ``, `` (real raw tag: ``) tags, `` blocks, and cross-tenant / Minutes / Tencent-Meeting URLs. Extract every reference, dispatch by type, fetch, and **repeat on each newly fetched doc until no new references remain** (leaf nodes) — **except `whiteboard`, which is never fetched or recursed**: it's inline visual content, not a link to another document (see the dedicated instruction below). Use the bundled extractor so nothing is silently missed (a missed reference = a missing document, the #1 hub-scraping failure): + +```bash +python3 scripts/feishu_extract_refs.py ".html" # → JSON list of {type, ref, title, dispatch} +``` + +Run this once per fetched document — the root, then every newly-fetched child — using that document's own `.html` (see step 3's naming caveat). The extractor is a plain regex scan — it works on either the `.html` or `.md` for a given document, since it just checks whether the file's text contains the tags — but **`.html` is the one to trust when both exist**; fall back to `.md` only if no `.html` was ever saved for that document (the `.data.markdown` branch, see step 3). Recursion loop, dispatch table, and the cross-tenant/`my.feishu.cn` personal-space rules are in the reference. -**4. If it's a collection/hub, follow the reference graph (BFS).** The hub body contains ``, ``, `` tags and cross-tenant / Minutes / Tencent-Meeting URLs. Extract every reference, dispatch by type, fetch, and **repeat on each newly fetched doc until no new references remain** (leaf nodes). Use the bundled extractor so nothing is silently missed (a missed reference = a missing document, the #1 hub-scraping failure): +**Whiteboard blocks are not a followable reference — export and read them in place.** A `whiteboard token="…"` tag is Feishu's native diagram/flowchart block, inlined in the current document — it is not a pointer to another document, so don't try to recurse/fetch it like `mention-doc`/`sheet`. It must be understood visually: a flowchart's meaning is not reliably recoverable from raw node coordinates/text-fragment lists alone. Export a preview image and actually look at it: ```bash -python3 scripts/feishu_extract_refs.py source.md # → JSON list of {type, token, title} +lark-cli whiteboard +export --whiteboard-token --output-type preview --output .jpg --overwrite ``` -Recursion loop, dispatch table, and the cross-tenant/`my.feishu.cn` personal-space rules are in the reference. +Then use the Read tool on the resulting `.jpg` to see the diagram's actual content. `--output-type raw` is also available (structured node JSON — useful as a cross-check/searchable index, but not a substitute for looking at the rendered image); `svg` and `source` output types also exist. The output path must have a real image extension matching what the command actually produces (it errors if you ask for `.png` when the true format is `.jpg` — match the extension to what the command reports, or omit the extension per its own `--help`). This matters because diagrams frequently carry decision-relevant content absent from the document's plain-text sections — e.g. a swimlane/process diagram can carry role-by-role steps and concrete numeric thresholds nowhere else in the doc. Treating a document as "fully extracted" without opening these blocks silently discards exactly the content most likely to carry its actual operating logic. -**5. Final residual-tag check (acceptance gate for collections).** Every rich-media reference must have been resolved and rendered: +**5. Final residual-tag check (acceptance gate — run this on every fetched document, not just collections).** A single standalone document with no cross-doc references still needs this: an inline `whiteboard` or unresolved reference tag can appear with zero other documents involved (this is exactly what happened on a real single-doc extraction 2026-08-16 — no hub, no recursion, just 3 unread whiteboards). Every rich-media reference must have been resolved and rendered. Run this recursively over the whole working directory, not a single file — a collection has one `.html`/`.md` pair per document (step 3), and the pandoc-converted `.md` can report "clean" on its own while real tags were silently dropped (see step 3's callout), so the scan needs to reach every `.html` on disk: ```bash -grep -rlE '<(lark-table|lark-tr|sheet token=|mention-doc|view type=)' . && echo "UNRESOLVED — keep recursing" || echo "clean" +grep -rlE '<(lark-table|lark-tr|sheet token=|mention-doc|cite doc-id=|whiteboard token=|view type=)' . \ + && echo "UNRESOLVED — keep recursing" || echo "clean" ``` -Must be empty before you stop. +`lark-tr` and `view type=` in that pattern are pre-existing, unverified-against-real-HTML terms — unlike the other five, they have no backing regex in `feishu_extract_refs.py` and no dispatch entry, so a hit here has no structured tooling support; treat it as "stop and inspect the raw tag by hand," not as something the extractor already understands. + +⚠️ **On each document's saved `.html`, "empty" is not a literal stop condition — treat each hit as a worklist item, not a failure to loop on.** Each `.html` is an immutable raw capture of *that* document (nothing in this skill rewrites a document's own file in place once fetched — see step 3's per-document naming caveat), so a parent doc that genuinely references N other docs or M diagrams keeps showing N+M matches forever in its own file, even after every one of them has been correctly handled — chasing this grep to a literal zero across the whole directory will never terminate for a real hub doc. Instead, for every match, verify the corresponding artifact exists on disk: a `mention-doc`/`cite doc-id=`/`sheet` hit is resolved once the referenced child doc has actually been fetched and saved (step 3, applied to that child); a `whiteboard` hit is resolved once its preview `.jpg` was exported and Read (step 4) — **never** by fetching another document, it is not a followable reference. Stop only once every match maps to a verified on-disk artifact. (On the `.data.markdown` fallback branch, where the fetched body is the deliverable Markdown itself rather than an immutable raw capture, a literal empty result remains the simpler signal — but that branch was not confirmed reachable on any current lark-cli build, see step 3.) ## Path B — permission denied → owner-exported .docx @@ -104,6 +129,7 @@ These are the rules whose violation silently ruins the output. Each has a reason - **Transcripts come from the platform's native transcription, never re-ASR.** Downloading media and transcribing again loses speaker labels, timestamps, and accuracy. - **A generated docx Markdown is not done until it has been *visually* verified** against the source (render to image, read it). Feishu-exported docx uses font-size+bold for headings rather than Word heading styles, so a "no errors, word count matches" check passes while the entire heading hierarchy is silently flat. Text-level checks cannot catch this. - **Do not 死磕 (grind) on docx embedded-image download.** lark-cli (through 1.0.32) cannot download `` tokens from a docx — exhaustively verified. Register the image tokens and note "needs document owner to right-click → save"; the text is the value, images are a tracked gap. +- **Rich-media tag verification must run on each document's own `.html`, never its `.md`, on the pandoc fallback path — and each document needs its own filename, not a shared literal `source.html`.** `pandoc -f html -t gfm` silently strips Feishu's custom embedded tags — verified on a real document: 3 raw `whiteboard token="…"` tags in `.data.document.content` left zero trace in the converted `.md` (2026-08-16). Checking only the `.md` for residual tags on this path always reports "clean," even when content was silently discarded; reusing one hardcoded filename across a hub's multiple fetches (step 3) would additionally let a later document silently overwrite an earlier one's raw capture before it was ever checked. - **HTTP 200 from anonymous curl ≠ accessible.** A Feishu login wall returns 200 with a body containing `accounts.feishu.cn` / `login` / `passport` / an empty ``. Check the body, never infer "public" from the status code. - **A file "not found" by a search agent is not authoritative.** Verify against authoritative sources before concluding (this is general Inference Discipline; relevant when locating where ingested content already lives). - **U+FFFD final check on every produced file:** `LC_ALL=C grep -rl $'\xef\xbf\xbd' .` must be empty. A replacement character means an encoding step corrupted the text. @@ -113,11 +139,11 @@ These are the rules whose violation silently ruins the output. Each has a reason Stop only when all that apply are true: - Every fetched body reached disk via `jq`/script, not retyped by the model. -- Collections: the residual rich-media-tag grep (Path A step 5) is empty — every `mention-doc`/`sheet`/cross-tenant reference was followed to a leaf. +- **Every fetched document — a lone doc as much as a collection**: every hit from the residual rich-media-tag check (Path A step 5, run recursively over the whole working directory) maps to a verified on-disk artifact — every `mention-doc`/`cite doc-id=`/`sheet`/cross-tenant reference was **followed** to a fetched leaf file, and every `whiteboard` reference was **exported and read** (not followed — a whiteboard is inline visual content, never a link to recurse into). This is not a collections-only check: a standalone document can contain an unresolved `whiteboard` with zero other documents involved. Each document's own `.html` legitimately keeps showing its tags forever (it's an immutable raw capture, never rewritten — as long as each document got its own filename per step 3) — don't chase the grep itself to a literal zero. - `LC_ALL=C grep -rl $'\xef\xbf\xbd' .` is empty. - docx path: rendered to an image and visually compared to the source; heading hierarchy and highlights match (see docx reference's checklist). - Browser fallback only: TOC coverage + scale check (see browser-failure-rules.md). -- Each output file's frontmatter records `source` (the original URL/token) and, if any post-processing was applied, a `post_process` provenance line. +- Each output file's frontmatter records `source` (the original URL/token) and, if any post-processing was applied, a `post_process` provenance line — the exact YAML shape and field list is **[references/lark-cli-api-extraction.md, Step 7](references/lark-cli-api-extraction.md)** (not shown in Path A's 5 numbered steps above, since it's a per-file finishing step rather than part of the fetch/recurse/check loop). - Permission gaps (131006 docs not exported yet, undownloadable images) are explicitly listed for the user — a transparent gap beats a silent omission. ## Do NOT attempt @@ -132,7 +158,7 @@ Verified dead-ends — retrying them only wastes the session. Full table with fa ## Bundled resources -- `scripts/feishu_extract_refs.py` — deterministic reference-token extractor; the recursion engine's core. Run it on every fetched body to enumerate `<mention-doc>`/`<sheet>`/`<image>`/cross-tenant/Minutes/Tencent-Meeting references as JSON. +- `scripts/feishu_extract_refs.py` — deterministic reference-token extractor; the recursion engine's core. Run it once per fetched document, on that document's own `<sanitized-title>.html` (prefer over `.md` — step 3), to enumerate `<mention-doc>`/`<sheet>`/`<image>`/`<whiteboard>`/cross-tenant/Minutes/Tencent-Meeting references as JSON. - `scripts/restore_docx_headings.py` — for Path B: reads true font sizes via python-docx, maps them to heading levels, restores `w:shd` highlights to Obsidian `==…==`, without retyping body text. - `scripts/feishu_dom_capture.js` — Path D: injectable end-to-end browser DOM capture. - `scripts/download_feishu_images.py` — Path D: SSR image extraction when browser automation is unavailable. diff --git a/feishu-doc-scraper/references/lark-cli-api-extraction.md b/feishu-doc-scraper/references/lark-cli-api-extraction.md index bf439493..4430afa1 100644 --- a/feishu-doc-scraper/references/lark-cli-api-extraction.md +++ b/feishu-doc-scraper/references/lark-cli-api-extraction.md @@ -74,15 +74,20 @@ The body field moved between lark-cli versions, so probe both instead of hard-co ```bash lark-cli docs +fetch --doc <obj_token> --format json > /tmp/fetch.json 2> /tmp/fetch.err # ≤1.0.32: clean Markdown in .data.markdown. -# 1.0.55: body moved to .data.document.content as HTML (.data.markdown is null). +# 1.0.55+: body moved to .data.document.content as HTML (.data.markdown is +# null — verified null in 11/11 real documents checked as of 2026-08-17, +# including 3/3 fresh fetches on the currently-installed 1.0.80; whether the +# .data.markdown branch is still reachable on any current build was not +# confirmed). if jq -e -r '.data.markdown // empty' /tmp/fetch.json > "<sanitized-title>.md" && [ -s "<sanitized-title>.md" ]; then : # got clean Markdown directly else - jq -r '.data.document.content' /tmp/fetch.json | pandoc -f html -t gfm > "<sanitized-title>.md" + jq -r '.data.document.content' /tmp/fetch.json > "<sanitized-title>.html" + pandoc -f html -t gfm "<sanitized-title>.html" > "<sanitized-title>.md" fi ``` -- On 1.0.55 the HTML in `.data.document.content` carries Feishu structure as `<h1>/<table>/<cite doc-id=…>` tags; `pandoc -f html -t gfm` renders headings and tables faithfully, and the `<cite>` tags surface as the reference list Step 5 recurses on. `--format markdown` is **not** a valid value (lark-cli warns and falls back to json). +- On 1.0.55+ the HTML in `.data.document.content` carries Feishu structure as `<h1>/<table>/<cite doc-id=…>` tags; `pandoc -f html -t gfm` renders headings and ordinary tables faithfully. ⚠️ **But `<cite>` (mention-doc) and `<whiteboard token=…>` do NOT survive — pandoc drops them with zero trace** (verified 2026-08-16/17 against real documents; for `<cite>` not even the bare title text survives, since the title lives in a `title="…"` attribute and the tag body is empty). Because of this, **Step 5's extractor and leaf-gate must run against the saved `<sanitized-title>.html`, never against the pandoc-converted `.md`**, whenever this branch was taken — see Step 5 below. `--format markdown` is **not** a valid value (lark-cli warns and falls back to json). - **Keep stdout/stderr separate.** `stderr` may carry `[deprecated] docs +fetch with v1 API is deprecated` — harmless. Doing `2>/dev/null | jq` in one pipe produced a spurious `Exit code 5`; redirect to files and inspect instead. - **Never** reconstruct the body by reading and retyping it — `jq`/`pandoc` it to disk. pandoc only re-renders HTML structure (it does not rewrite prose), so the fidelity guarantee that makes Path A safer than any browser/LLM path still holds. - `--format json` is required so you parse one field deterministically. @@ -140,11 +145,13 @@ A hub is the root of a reference graph. Treat it as BFS/DFS over references unti **Enumerate references with the bundled extractor** (a missed reference is a missing document — the single biggest hub-scraping failure; do not hand-roll `grep` and forget the `my.feishu.cn` personal-space pattern, which is exactly what happened before this script existed): ```bash -python3 scripts/feishu_extract_refs.py <fetched-body>.md -# → JSON array of {type, token_or_url, title} +python3 scripts/feishu_extract_refs.py <fetched-body>.html +# prefer the raw HTML saved in Step 3; fall back to <fetched-body>.md only if +# no .html was ever saved (the .data.markdown branch) +# → JSON array of {type, ref, title, dispatch} ``` -The references it recognizes (the full rich-media inventory): `<mention-doc token type>`, `<sheet token>`, `<lark-table><lark-tr><lark-td>` (inline tables — render in place, not a reference), `<image token>`, `<view><file>`, cross-tenant `https://<tenant>.feishu.cn/(docx|wiki|sheets|base|file)/<token>`, personal-space `https://my.feishu.cn/docx/<token>`, Minutes `https://<tenant>.feishu.cn/minutes/<token>`, Tencent-Meeting `https://meeting.tencent.com/crm/<id>`. +The references it recognizes (the full rich-media inventory): `<mention-doc token type>`, `<sheet token>`, `<lark-table><lark-tr><lark-td>` (inline tables — render in place, not a reference), `<image token>`, `<whiteboard token>` (inline diagram block — export+Read, NOT a followable reference, see below), `<view><file>`, cross-tenant `https://<tenant>.feishu.cn/(docx|wiki|sheets|base|file)/<token>`, personal-space `https://my.feishu.cn/docx/<token>`, Minutes `https://<tenant>.feishu.cn/minutes/<token>`, Tencent-Meeting `https://meeting.tencent.com/crm/<id>`. ⚠️ The `<mention-doc>`/`<image>`/`<lark-table>` shapes above are this doc's shorthand names for the reference *types*, not their literal raw-HTML syntax — the real, verified tag shapes (`<cite doc-id=…>`, `<img src=…>`, plain `<table>`) are documented with evidence in `scripts/feishu_extract_refs.py`'s regex comments; trust those over these shorthand names when hand-verifying a residual tag. **Dispatch table:** @@ -157,6 +164,7 @@ The references it recognizes (the full rich-media inventory): `<mention-doc toke | `meeting.tencent.com/crm/` | Tencent Meeting tooling (outside this skill — its native transcript API; never download+re-ASR) | | `<lark-table>` | render inline to a Markdown table (pandas `read_html` handles colspan/rowspan); it is content, not a link | | `<image token>` | register the token; lark-cli cannot download it (see permission-and-failure-boundaries.md) | +| `<whiteboard token>` | NOT recursed — export a preview image (`lark-cli whiteboard +export --whiteboard-token <token> --output-type preview --output <path>.jpg --overwrite`) and Read it; see SKILL.md Path A step 4 | | `<view><file>` | attachment — record token + filename; treat like an image gap unless separately retrievable | **Recursion loop:** fetch root → extract refs → for each new ref, dispatch and fetch → run the extractor on each newly fetched body → repeat until no new tokens appear. A child doc can itself embed another reference (e.g. a summary doc that embeds a third Minutes link); the loop must re-scan every newly fetched file, not only the root. @@ -164,11 +172,11 @@ The references it recognizes (the full rich-media inventory): `<mention-doc toke **Leaf / completion gate** — before declaring the collection done, no rich-media reference may remain unresolved anywhere: ```bash -grep -rlE '<(lark-table|lark-tr|sheet token=|mention-doc|view type=)' . \ +grep -rlE '<(lark-table|lark-tr|sheet token=|mention-doc|cite doc-id=|whiteboard token=|view type=)' . \ && echo "UNRESOLVED — keep recursing" || echo "clean" ``` -This grep being empty is a hard acceptance gate for collections. +⚠️ **On each document's saved `.html`, "empty" is not a literal stop condition.** Each `<sanitized-title>.html` (Step 3's per-document naming — never a literal shared `source.html` across a hub's multiple fetches, or the second fetch silently destroys the first) is an immutable raw capture of that one document — nothing in this skill rewrites a document's own file in place — so a hub doc that genuinely references N children or M diagrams keeps showing N+M matches forever in its own file, even after every one is correctly handled; chasing this grep to a literal zero across the whole directory will never terminate for a real hub. Treat each match as a worklist item: a `mention-doc`/`cite doc-id=`/`sheet` hit is resolved once the referenced child has actually been fetched and saved (as its own distinct file); a `whiteboard` hit is resolved once its preview `.jpg` was exported and Read (dispatch-table row above) — it is not a missing document to fetch, don't treat it as one. This grep still recurses over every file already on disk, so it naturally covers every document's raw `.html` saved in Step 3 alongside its pandoc-converted `.md` — no extra step needed there. (On the `.data.markdown` branch, where the fetched body is the deliverable Markdown itself, a literal empty result remains the simpler signal.) ## Step 6: cross-tenant and personal-space sources diff --git a/feishu-doc-scraper/scripts/feishu_extract_refs.py b/feishu-doc-scraper/scripts/feishu_extract_refs.py index 9c97cc20..eacfaf50 100644 --- a/feishu-doc-scraper/scripts/feishu_extract_refs.py +++ b/feishu-doc-scraper/scripts/feishu_extract_refs.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Enumerate every rich-media reference in a fetched Feishu Markdown body. +"""Enumerate every rich-media reference in a fetched Feishu document body. This is the recursion engine's core for Path A (lark-cli API extraction). A collection/hub is a doc whose body references other docs; missing one @@ -7,17 +7,36 @@ Hand-rolled `grep | sed` pipelines repeatedly missed the `my.feishu.cn` personal-space pattern, so this enumeration is centralized and tested here. -Input : a Markdown file produced by `lark-cli docs +fetch ... | jq -r .data.markdown`. +Covers: mention-doc (real raw tag is `<cite doc-id=... file-type=... +title=...>`, verified 2026-08-17 — see RE_CITE_TAG), sheet, image (real raw +tag is `<img src=... alt=...>`, verified 2026-08-17 — see RE_IMG_TAG), file, +whiteboard (`<whiteboard token=...>` — an inline diagram block, NOT a +followable reference, see DISPATCH["whiteboard"]), lark-table, and +cross-tenant / personal-space / Minutes / Tencent-Meeting URLs. + +Input : the fetched Feishu body. On lark-cli builds where `.data.markdown` is + non-null (<=1.0.32; unconfirmed whether still reachable on any + current build, see SKILL.md Path A step 3), that Markdown is a valid + input. Otherwise (`.data.markdown` is null -- the current default, + verified null in 11/11 real documents checked including the + currently-installed lark-cli 1.0.80) the body only reaches disk as + raw HTML via `.data.document.content`; run this script on THAT saved + `source.html`, never on the pandoc-converted `source.md` -- pandoc + silently strips several of these tags with zero trace (whiteboard + and mention-doc confirmed; see SKILL.md Path A step 3's callout). Output: JSON array on stdout, one object per *distinct* reference: {"type": ..., "ref": <token-or-url>, "title": ..., "dispatch": <hint>} plus a human summary on stderr. It only *enumerates*. Dispatching/fetching each reference is the caller's job -(see references/lark-cli-api-extraction.md, Step 5 dispatch table). +(see references/lark-cli-api-extraction.md, Step 5 dispatch table) -- except +`whiteboard`, which is never fetched or recursed, only exported and read as +an image (see SKILL.md Path A step 4). Usage: - python3 feishu_extract_refs.py FETCHED_BODY.md - python3 feishu_extract_refs.py FETCHED_BODY.md --type docx # filter + python3 feishu_extract_refs.py FETCHED_BODY.html # prefer the raw HTML + python3 feishu_extract_refs.py FETCHED_BODY.md # only if no .html was saved + python3 feishu_extract_refs.py FETCHED_BODY.html --type docx # filter """ from __future__ import annotations @@ -32,13 +51,111 @@ # /minutes /base /file path scheme. _HOST = r"[a-z0-9-]+\.(?:feishu\.cn|larksuite\.com)" -# Inline rich-media tags emitted by `docs +fetch` Markdown. +# Inline rich-media tags. Two eras coexist here: the pre-1.0.55 `.data.markdown` +# pseudo-tags this script originally assumed (RE_MENTION_DOC, RE_IMAGE_TAG -- +# never observed in real raw HTML, but retained since nothing was verified +# about what that branch emits when it fires) and the verified 1.0.55+ raw +# `.data.document.content` HTML shapes (RE_CITE_TAG, RE_IMG_TAG) that replace +# them on the current default (pandoc) path. Run this script on source.html +# for the latter -- see the module docstring. + +# --- mention-doc references ------------------------------------------------- +# Verified 2026-08-17 (lark-cli 1.0.80, fresh fetch of a real hub doc; the +# same byte-identical tag structure independently corroborated in an +# archived 2026-07-25 fetch of a different doc -- 6 weeks apart, same +# doc-id) against real `.data.document.content` raw HTML: the tag is NOT +# `<mention-doc token="..." type="...">Title</mention-doc>`. It is an +# empty-bodied <cite> tag with the title in an ATTRIBUTE, not inner text: +# <cite doc-id="FRigwwIkWiy5PSkBNiMcwN98nsY" file-type="wiki" +# title="手工记录反馈维度" type="doc"></cite> +# Confirmed file-type values: "wiki", "docx" (both independently verified +# against real documents). `pandoc -f html -t gfm` makes this tag VANISH +# WITH ZERO TRACE -- not even the bare title text survives (worse than +# losing just the token/type) -- so this pattern is only useful against raw +# HTML (source.html), never against a pandoc-converted .md. +RE_CITE_TAG = re.compile(r'<cite\b(?=[^>]*\bdoc-id=)[^>]*>') +_ATTR_DOC_ID = re.compile(r'\bdoc-id="([^"]*)"') +_ATTR_FILE_TYPE = re.compile(r'\bfile-type="([^"]*)"') +_ATTR_TITLE = re.compile(r'\btitle="([^"]*)"') + +# Legacy `<mention-doc token="..." type="...">Title</mention-doc>` pseudo-tag. +# Never observed in raw `.data.document.content` HTML across 11 real +# documents checked (2026-08-17) -- but nothing was verified about what the +# <=1.0.32 `.data.markdown` branch emits when it fires (no non-null +# `.data.markdown` response was found to inspect). Retained, not removed, as +# that branch's only coverage; costs zero false positives since this literal +# shape does not occur in raw HTML. RE_MENTION_DOC = re.compile( r'<mention-doc\s+token="([^"]+)"\s+type="([^"]+)"\s*>([^<]*)</mention-doc>' ) + +# --- sheet references -------------------------------------------------- +# UNVERIFIED -- no real document containing an actual sheet/spreadsheet +# reference was found (9 real documents inspected, 2026-08-17). Left +# unchanged: do not guess-fix a pattern with no positive evidence. If sheet +# references turn out to use the same <cite file-type="sheet"> shape as +# mention-doc (a plausible but unconfirmed extrapolation), the +# "mention-doc-other" bucket below will still surface them with the real +# file-type value visible for manual triage, rather than silently +# mis-tagging them as a confirmed fact. RE_SHEET_TAG = re.compile(r'<sheet\s+token="([^"]+)"\s*/?>') + +# --- image references ------------------------------------------------------- +# Verified 2026-08-17 (fresh fetch of a real docx, byte-identical to an +# archived 2026-07-25 fetch of the same doc) against real raw HTML: the tag +# is NOT `<image token="...">`. It is a standard-shaped self-closing <img> +# element with the drive token in `src=`, not `token=`: +# <img id="..." name="filename.jpeg" alt="<long AI-generated description>" +# height="1920" href="https://internal-api-drive-stream.feishu.cn/..." +# mime="image/jpeg" scale="0.288889" src="TP29bjDJSoUsilxKjiLcGi3UnTh" +# width="1080"/> +# Unlike mention-doc, this does NOT vanish under pandoc: `pandoc -f html -t +# gfm` re-serializes the whole <img> as raw HTML passthrough (src/id/href/ +# width/height/alt survive verbatim; `name=` is dropped; `mime=`/`scale=` are +# renamed `data-mime=`/`data-scale=` -- observed once cleanly, not proven +# universal across pandoc versions). So this pattern also works against a +# pandoc-converted source.md in practice, but source.html remains the one to +# trust (unlike RE_CITE_TAG above, which does NOT survive pandoc at all). +RE_IMG_TAG = re.compile(r'<img\b(?=[^>]*\bsrc=)[^>]*>') +_ATTR_SRC = re.compile(r'\bsrc="([^"]*)"') + +# Legacy `<image token="...">` pseudo-tag. Never observed in raw HTML (same +# caveat as RE_MENTION_DOC above: retained as the <=1.0.32 `.data.markdown` +# branch's only coverage, not removed). RE_IMAGE_TAG = re.compile(r'<image\s+token="([^"]+)"') + +# NOT independently researched (out of the 2026-08-17 investigation's +# scope). Left unchanged pending real evidence, same posture as +# RE_SHEET_TAG above. RE_FILE_TAG = re.compile(r'<file\s+token="([^"]+)"[^>]*>([^<]*)</file>') + +# --- whiteboard (inline diagram, NOT a followable reference) ---------------- +# Verified 2026-08-16 against a real document (3 tags in raw HTML, 0 trace +# in the pandoc-converted .md of the same doc). This is Feishu's native +# diagram/flowchart block, inlined in the current doc -- it is never +# fetched/recursed like mention-doc/sheet; see DISPATCH["whiteboard"] and +# SKILL.md Path A step 4 for the export+Read handling. +RE_WHITEBOARD = re.compile(r'<whiteboard\s+token="([^"]+)"') + +# --- inline tables ----------------------------------------------------- +# Verified 2026-08-17: ordinary docx tables use plain, standard HTML5 +# `<table>` markup, NOT a custom `<lark-table>` tag (0 occurrences of +# `<lark-table` across 9 real documents inspected). Contrary to the original +# assumption, pandoc does NOT collapse them into unstructured run-together +# text: tables whose cells hold single-paragraph content convert cleanly to +# GFM pipe-table syntax; tables with multi-paragraph cells fall back to a +# single embedded raw `<table>...</table>` HTML block (still structurally a +# table, just not pipe-syntax) -- observed in one real document (8 tables), +# not proven universal (colspan/rowspan not present in that sample). Because +# ordinary tables are common and mostly survive intact, this regex is +# deliberately NOT repointed at bare `<table` -- every healthy document has +# `<table>` tags, so that would turn the acceptance-gate check (SKILL.md +# Path A step 5) into constant false positives on completely healthy +# extractions. Left matching the literal (and, per this evidence, likely +# nonexistent-in-this-form) `<lark-table` tag, which costs nothing since it +# was already never observed to fire on real content; it may name an +# embedded Bitable/Base-view object not yet encountered, or may not exist in +# the current Feishu docx export format at all. RE_LARK_TABLE = re.compile(r"<lark-table\b") # URLs that appear in the body (cross-tenant / personal space / minutes / @@ -56,7 +173,8 @@ DISPATCH = { "mention-doc-docx": "docs +fetch --doc <token>", "mention-doc-wiki": "wiki spaces get_node then docs +fetch", - "mention-doc-sheet": "sheets +read", + "mention-doc-sheet": "sheets +read", # not currently produced -- see mention-doc-other; kept for when sheet's real <cite> shape is confirmed + "mention-doc-other": "unconfirmed file-type on a real <cite> tag — inspect manually (sheet/bitable is a plausible but UNVERIFIED guess, see RE_SHEET_TAG comment)", "url-docx": "docs +fetch --doc <token>", "url-wiki": "wiki spaces get_node then docs +fetch", "url-sheets": "sheets +read (split token on '_' -> SP, SID)", @@ -68,6 +186,7 @@ "file": "attachment — record token + name; treat like image gap", "tencent-meeting": "Tencent Meeting native transcript (never download+re-ASR)", "lark-table": "inline content — render in place to a Markdown table", + "whiteboard": "export preview + Read image (see SKILL.md Path A step 4) — NOT a followable reference", } @@ -93,13 +212,16 @@ def _read_text(path: Path) -> str: sys.exit( f"error: {path} is not valid UTF-8 ({exc}); an upstream extraction " f"step corrupted the body — re-fetch with `lark-cli docs +fetch " - f"--format json` and `jq -r .data.markdown`, do not 'fix' encoding here." + f"--format json` and re-derive source.html/.md per SKILL.md Path A " + f"step 3, do not 'fix' encoding here." ) def extract(text: str) -> list[dict]: refs: list[dict] = [] + # Legacy `.data.markdown`-era pseudo-tag (never observed in real raw + # HTML — see RE_MENTION_DOC comment). Retained for that branch only. for token, doc_type, title in RE_MENTION_DOC.findall(text): t = doc_type.strip().lower() kind = "mention-doc-sheet" if t in ("sheet", "bitable") else ( @@ -112,6 +234,36 @@ def extract(text: str) -> list[dict]: "dispatch": DISPATCH[kind], }) + # Verified 1.0.55+ raw-HTML shape (see RE_CITE_TAG comment above). This + # is what actually fires against real source.html today. + for tag in RE_CITE_TAG.findall(text): + doc_id_m = _ATTR_DOC_ID.search(tag) + if not doc_id_m: + continue # lookahead guarantees doc-id is present; stay defensive + doc_id = doc_id_m.group(1) + file_type_m = _ATTR_FILE_TYPE.search(tag) + file_type = file_type_m.group(1).strip().lower() if file_type_m else "" + title_m = _ATTR_TITLE.search(tag) + title = title_m.group(1).strip() if title_m else "" + + if file_type == "wiki": + kind, dispatch = "mention-doc-wiki", DISPATCH["mention-doc-wiki"] + elif file_type == "docx": + kind, dispatch = "mention-doc-docx", DISPATCH["mention-doc-docx"] + else: + # Includes the unverified "sheet"/"bitable" hypothesis (see + # RE_SHEET_TAG comment above) and any other value — surfaced for + # manual triage rather than guess-classified as a confirmed fact. + kind = "mention-doc-other" + dispatch = f"unconfirmed file-type={file_type!r} on a real <cite> tag — inspect manually" + + refs.append({ + "type": kind, + "ref": doc_id, + "title": title, + "dispatch": dispatch, + }) + for token in RE_SHEET_TAG.findall(text): refs.append({ "type": "sheet-tag", @@ -120,6 +272,8 @@ def extract(text: str) -> list[dict]: "dispatch": DISPATCH["sheet-tag"], }) + # Legacy `.data.markdown`-era pseudo-tag (never observed in real raw + # HTML — see RE_IMAGE_TAG comment). Retained for that branch only. for token in RE_IMAGE_TAG.findall(text): refs.append({ "type": "image", @@ -128,6 +282,27 @@ def extract(text: str) -> list[dict]: "dispatch": DISPATCH["image"], }) + # Verified 1.0.55+ raw-HTML shape (see RE_IMG_TAG comment above). This + # is what actually fires against real source.html/source.md today. + for tag in RE_IMG_TAG.findall(text): + src_m = _ATTR_SRC.search(tag) + if not src_m: + continue # lookahead guarantees src is present; stay defensive + refs.append({ + "type": "image", + "ref": src_m.group(1), + "title": "", + "dispatch": DISPATCH["image"], + }) + + for token in RE_WHITEBOARD.findall(text): + refs.append({ + "type": "whiteboard", + "ref": token, + "title": "", + "dispatch": DISPATCH["whiteboard"], + }) + for token, name in RE_FILE_TAG.findall(text): refs.append({ "type": "file", @@ -175,7 +350,7 @@ def extract(text: str) -> list[dict]: def main() -> None: ap = argparse.ArgumentParser(description=__doc__) - ap.add_argument("markdown_file", help="fetched Feishu body (.md)") + ap.add_argument("markdown_file", help="fetched Feishu body (.html preferred, .md if no .html was saved)") ap.add_argument("--type", help="only emit refs of this type (e.g. docx, image)") args = ap.parse_args()