fix(feishu-doc-scraper): acceptance gate must check raw source.html, not pandoc-stripped source.md - #302
Merged
Merged
Conversation
… source.md, not raw source.html pandoc -f html -t gfm silently drops several of Feishu's real embedded-content tags when converting .data.document.content (the current default fetch path, since .data.markdown is null on every real document checked, incl. lark-cli 1.0.80). The skill's own residual-tag acceptance gate (Path A step 5) grepped the post-pandoc source.md, so it reported "clean" even when real references were silently discarded -- confirmed live tonight: a real document with 3 unread <whiteboard> diagram blocks passed the old gate as "clean". Real raw-HTML tag structure was verified against 11 real documents (not assumed from a synthetic test) and turned out to differ from what the extractor originally assumed for every type checked: - mention-doc's real tag is <cite doc-id=... file-type="wiki|docx" title="...">, not <mention-doc token=... type=...>Title</mention-doc> -- and it vanishes under pandoc with zero trace (title lives in an attribute, not inner text). - image's real tag is a standard <img src=... alt=...>, not <image token=...> -- and it mostly survives pandoc as raw-HTML passthrough. - lark-table isn't a real custom tag for ordinary docx tables at all -- they use plain HTML <table>, which pandoc converts cleanly. Changes: - Path A step 3 now always saves source.html to disk before the pandoc conversion (both files exist afterward), with a callout documenting the per-tag pandoc-survival findings above. - Path A steps 4/5 (extraction + residual-tag check) now operate on source.html, never source.md, on the pandoc path. - The residual-tag check gate is no longer collections-only -- a standalone document (no cross-doc references) can still contain an unresolved inline whiteboard, which is exactly tonight's real failure mode. - Clarified that source.html is an immutable raw capture that legitimately keeps showing resolved tags forever; the gate now checks that each hit maps to a verified on-disk artifact, not that the grep goes literally empty. - feishu_extract_refs.py: added RE_CITE_TAG and RE_IMG_TAG for the verified real tag shapes (with attribute extraction for doc-id/file-type/title), added whiteboard detection, kept the old assumed patterns as untested fallback coverage for the (possibly dead) <=1.0.32 .data.markdown branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QAhkXwgkjK91hFpJLhrLQ
…l/.md would collide across a hub's recursive fetches Independent review of the previous commit (bb9e974) found a real internal contradiction via live replay: Path A step 3's condensed snippet hardcoded a literal source.html/source.md filename for every fetch. Step 4 explicitly recurses into every referenced child document, fetching each one into the same working directory -- so a second fetch would silently overwrite the first document's saved raw HTML before anyone checked it. This directly falsified the "source.html is an immutable raw capture, never rewritten" claim the previous commit had just added. references/lark-cli-api-extraction.md already used a correct, per-document `<sanitized-title>` filename pattern for the same procedure -- SKILL.md's condensed quick-start had just never been kept in sync with it. Fix: step 3 now templates the output filename per document; step 5's residual-tag check is now a recursive whole-directory grep (matching the reference file) instead of a single-file check, so it can find and name which specific document still has an unresolved tag across a multi-document collection. Surrounding prose (step 4's invocation, the Hard Rules bullet, the Acceptance Contract bullet) updated to stop implying "source.html" is one shared filename. Verified by replay: simulated a 2-fetch hub scrape (parent + child) using the new step 3 pattern -- the parent's unique marker text survives in both its .html and .md after the child fetch, and the new recursive step-5 gate correctly names the parent's file as the one with the unresolved reference. Also closes two smaller, pre-existing gaps the same review surfaced: - Acceptance Contract required frontmatter but no numbered step pointed to where the field list actually is (references/..., Step 7) -- added the pointer. - The path-selection flowchart listed "base" (Bitable) as an in-scope source type, but Path A never operationalizes it (the extractor only records the token) -- added an explicit caveat so a reader isn't misled into expecting end-to-end Bitable extraction from Path A. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QAhkXwgkjK91hFpJLhrLQ
…b9e974's tag-shape fix A follow-up confirmation pass on c31a2a0 (the per-document filename fix) flagged two spots -- SKILL.md's Bundled resources line and the reference file's leaf-gate warning paragraph -- that still said "source.html" as if it were a single shared filename, unqualified by the per-document naming established in step 3. Both predate this fix (introduced in bb9e974, the original tag-shape correction) and are low severity / non-functional -- the actual overwrite risk lives entirely in step 3's fetch code, which was already correctly fixed everywhere by c31a2a0 -- but they're a terminology inconsistency worth closing for a reader skimming just these two spots. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QAhkXwgkjK91hFpJLhrLQ
daymade
added a commit
that referenced
this pull request
Aug 17, 2026
- feishu-doc-scraper 1.3.2→1.3.3: #302 验收闸门修复(raw source.html 检查) 漏了强制版本 bump, 补记 CHANGELOG - daymade-skill 1.25.0→1.25.1: --renamed-from 修复(bb300a0+bd539d9)同样 漏 bump, 补记 CHANGELOG - transcript-fixer 1.24.0→1.24.1: main 上虚构语境里残留的「1v1沟通」 token 换成「1页纸汇报」——单看是通用商务用语不算泄露, 但与已脱敏的 真实项目指纹同源可辨, 顺手换干净; 保留测试需要的孤位数字形状, 22/22 测试过 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
source.md.pandoc -f html -t gfmsilently strips several of Feishu's real embedded-content tags when converting.data.document.content(the current default fetch path —.data.markdownis null on every real document checked, including lark-cli 1.0.80), so the gate reported "clean" even when real references were silently discarded. Confirmed live: a real document with 3 unread<whiteboard>diagram blocks passed the old gate as "clean".feishu_extract_refs.pyoriginally assumed for every type checked — see the commit message for the per-tag findings (<cite doc-id=...>not<mention-doc token=...>, standard<img src=...>not<image token=...>, plain<table>not a custom<lark-table>).Test plan
scripts/quick_validatepasses on the edited skillast.parseon the edited script — no syntax errorssource.md) reports "clean"; new (checkingsource.html) reports "UNRESOLVED" — live reproduction of the bug and the fixsource.htmlis never rewritten, and the gate was scoped to collections-only)🤖 Generated with Claude Code
https://claude.ai/code/session_016QAhkXwgkjK91hFpJLhrLQ