[docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49)

[kiki830621]

Problem

From verification of #49:
「YAML message_id: parser brittle for single-quoted / inline-comment / CRLF / trailing-whitespace cases. archive-mail v2.6+ self-archived files always double-quoted (19 chchen_lab historical files all OK), so impact limited to manual-archive edge case. README claim "Resilient to schema diversity" is overpromised.」
— Source: idd-verify #49 (logic F5 + codex P3 + DA verification)

archive-mail.md Step 2.1 sibling-archive dedup extension uses awk to extract message_id: value from markdown YAML frontmatter:

mid=$(head -30 "$mdfile" 2>/dev/null | awk '/^message_id:[ \t]*/{sub(/^message_id:[ \t]*"?/,"");sub(/"?$/,"");print;exit}')

This works for archive-mail's own writer (always emits message_id: "<...>" with double quotes), but the README and docs claim "Resilient to schema diversity" — overpromise.

Type

docs (claim reduction) OR refactor (parser hardening)

Three sub-issues

A. False claim in archive-mail.md properties block

archive-mail.md Step 2.1 properties states "Resilient to schema diversity:只 require message_id: field 在 YAML frontmatter 開頭~30 lines". This is not actually robust — it fails on:

• Single-quoted values: message_id: '<...>'
• Inline comments: message_id: "<...>" # received 2026-01-01
• CRLF line endings (sub trailing ?$/ matches \r")
• Trailing whitespace before closing quote
• Multi-line YAML values (rare but valid)

B. Mitigations

Two paths:

1. Reduce claim: change "Resilient to schema diversity" to "Compatible with archive-mail's own message_id frontmatter format (v2.6+); manual archives or external sources may need explicit conversion". Doc-only fix, pure honesty.

2. Strengthen parser: switch from awk to a proper YAML parser (e.g. yq, or hand-rolled awk that handles quotes / comments / CRLF). Requires yq dep OR more bash code.

C. README log line claim

README L99 says detection "extends dedup with N entries" — if YAML parser silently skips files, N undercounts without warning. User can't tell if dedup is actually working from the log.

Priority

P3 — low impact unless user has manually-edited or externally-sourced markdown archives. archive-mail's own outputs are 100% covered by the current parser.

Strategy sketch

Recommended: Option A.1 (claim reduction) for cheap fix. Defer parser hardening until N≥3 user reports of manual-archive missed dedup.

- - **Resilient to schema diversity**:只 require `message_id:` field 在 YAML frontmatter 開頭~30 lines;archive-mail v2.6.0+ 全有,manual archive 若有 `message_id:` 也認得;沒有則跳過該檔(silent skip,計入 ENTRIES_THIS_DIR=0 不報)。
+ - **archive-mail v2.6+ format compatibility**:設計 target 是 archive-mail 自產的 frontmatter(`message_id: "<...>"` 雙引號格式)。Manual archive 若 frontmatter 用單引號 / 內含 inline comment / CRLF / trailing whitespace,parser 會 silent skip 該檔(計入 ENTRIES_THIS_DIR=0)。Future hardening 見 follow-up issue。

Plus add a runtime log when ENTRIES_THIS_DIR=0 but *.md files exist in symlink dir (signal silent-skip to user).

Related

• archive-mail: support workspaces with nested communications layout and symlinked historical archives #49 (parent)
• [refactor] Apply same detection cascade to documents_dir as output_dir (sister concern from #49) #50 (sibling, may have similar parser hardening question for documents_dir)

Current Status

Phase: implemented (cluster-PR mode)
Last updated: 2026-05-10 by /idd-implement cluster

Key Decisions

• Part of 4-issue docs cluster ([docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52 + [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 + [docs] README workspace pattern '100% backward compat' wording too strong (sister concern from #49) #55 + [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64) sharing PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 + cluster branch idd/cluster-docs-followups-49-60.
• Implementation: archive-mail.md schema-diversity claim reduction.
• Commit: 70e4485 references this issue via Refs PsychQuant/psychquant-claude-plugins#54.
• Pure docs change — no code logic, no test impact, no version bump.

Scope Changes

• Single file change per issue, all 4 issues touch different files (no inter-issue scope overlap).
• No sister bugs surfaced (Step 5.7 sweep returned empty).

Blocking

• (none) Awaiting /idd-verify --pr 71 cluster verify + per-issue /idd-close after merge.

Commits

• 70e4485: archive-mail.md schema-diversity claim reduction (Refs [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54)
• (cluster total: 4 commits — see PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 for full diff)

Current Status

Phase: verified
Last updated: 2026-05-10 by /idd-verify cluster

Key Decisions

• Cluster verify (PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71) returned PASS after empirical adjudication of Codex GPT-5.5 cross-model dissent. Full report: cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 (comment)
• This issue's verify-stage fix(es): MEDIUM parser failure-mode wording corrected
• Codex caught parser-behavior accuracy issue Claude reviewer flat-passed — cross-model verify did its job.
• 2 Codex HIGH findings on ### Notes non-KAC section names rejected as false positives — empirical evidence shows ### Notes is established repo convention (6× usage in che-apple-mail-mcp CHANGELOG, also batch-ocr/plugin-tools/parallel-ai-agents).
• DA skipped — disagreements were empirically resolvable via repo grep + bash awk test, no need for separate DA agent.

Scope Changes

• (none from this issue's perspective; cluster-level only)
• Cross-issue observation filed inline in master report: plugin currently v2.18.1 but CHANGELOG missing [2.18.1] entry (same KAC drift class as [docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52) — suggested follow-up for next cluster.

Blocking

• (none) Awaiting PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 merge + per-issue /idd-close closing summary.

Commits

• c04f66d: fix(docs-cluster): apply /idd-verify PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 codex findings (Refs [docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52 [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64)
• (this-issue commit on cluster branch): see PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 for full diff

Current Status

Phase: closed
Last updated: 2026-05-10 by /idd-close cluster

Key Decisions

• Closed via cluster /idd-close PsychQuant/psychquant-claude-plugins#52 PsychQuant/psychquant-claude-plugins#54 PsychQuant/psychquant-claude-plugins#55 PsychQuant/psychquant-claude-plugins#64 after PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 merged.
• Per-issue closing summary written independently (NOT batched — IDD discipline).
• Step 0 supersession active(Implementation Complete > Checklist all - [x])→ Strategy/Plan unchecked items 視為 superseded per #515 fix。
• Step 1.5 PR Gate PASS(PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 MERGED at 2026-05-10T12:02:27Z as 1557fbf)。
• Step 1.6 semantic gate PASS(每個 issue 的 backtick file path 都在 #54 commit 的 changed-files 列表)。

Scope Changes

• Cluster of 4 issues sharing 1 PR + 1 squash-merge commit 1557fbf(原 5 commits squashed: 4 implementation + 1 verify-fix)。
• Verify-stage findings:cross-model dissent caught accuracy issues Claude flat-passed(parser-behavior wording, scope claim narrowing, suppression description completeness)。
• 2 Codex HIGH on ### Notes non-KAC rejected as repo-convention false positives。

Blocking

• (none — issue CLOSED)

Commits

• 1557fbf: cluster: 4 docs follow-ups from archive-mail: support workspaces with nested communications layout and symlinked historical archives #49 / plugin-update: add Step 0 git-state preview + commit/push confirmation dialog before mutating origin #60 IDD lifecycles (cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71) — squash merge
• (Pre-squash on cluster branch idd/cluster-docs-followups-49-60,already deleted post-merge)

[kiki830621]

Implementation Plan (cluster-PR mode)

This issue is part of a 4-issue docs cluster sharing 1 feature branch + 1 PR. Plan covers all 4 issues for transparency; commits use Refs #N to match each item to its source issue.

Cluster scope:

• [docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52 — Backfill CHANGELOG [2.16.1] entry in che-apple-mail-mcp
• [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 — Reduce "Resilient to schema diversity" claim in archive-mail.md Step 2.1
• [docs] README workspace pattern '100% backward compat' wording too strong (sister concern from #49) #55 — Soften "100% backward compat" wording in che-apple-mail-mcp/README.md
• [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64 — Backfill [1.15.0] placeholder date in plugin-tools/CHANGELOG.md + restore description detail

Branch: idd/cluster-docs-followups-49-60

Files & Changes (per-issue)

#52 (commit 1)

• plugins/che-apple-mail-mcp/CHANGELOG.md — insert ## [2.16.1] - 2026-05-09 block between current [2.17.0] and [2.16.0]. Section: ### Notes (bump-only). Content: ship binary v2.7.1 (cluster: 4 refactor/docs follow-ups from #49 / #60 IDD lifecycles #72 base64 fix + [refactor] Define IDD_ALL_UNATTENDED env var contract across idd-all + plugin-tools (sibling concern from #60) issue-driven-development#123 SQLite fast-path stderr logging + [enhancement] che-apple-mail-mcp session-start hook: PID validation + started_at comparison hardening (follow-up from #76) #66 .partial.emlx attachment fix) + tool count drift catch-up 44→47.

#54 (commit 2)

• plugins/che-apple-mail-mcp/commands/archive-mail.md — Step 2.1 properties block, replace "Resilient to schema diversity" claim with v2.6+ format-compatibility wording matching issue body's diff sketch. Pure docs (option A.1 per issue), no parser hardening (deferred).

#55 (commit 3)

• plugins/che-apple-mail-mcp/README.md — Workspace Patterns section, replace "100% backward compat" wording with explicit-pin / mid-migration distinction per issue body's diff sketch.

#64 (commit 4)

• plugins/plugin-tools/CHANGELOG.md — replace ## [1.15.0] - (date unknown — please fill in) placeholder with ## [1.15.0] - 2026-04-26. Backfill description per scope expansion: README Freshness Gate v2 (6 signals + 2 suppressions), component inventory drift / tool count drift / version history multi-version gap, cross-28-plugin audit.

Sequencing

4 disjoint files → 4 independent commits, any order. Choose simplest visual order: #52 → #54 → #55 → #64.

Decision Points (locked in)

• D1 — Cluster vs separate PRs: chose cluster because all 4 are pure docs, low review surface, batch-friendly. Separate PRs would create review fragmentation for marginal benefit.
• D2 — [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 scope: claim reduction only (Option A.1 per issue body); no parser hardening this PR. Defer until N≥3 user reports of manual-archive missed dedup.
• D3 — [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64 scope: include description backfill (per scope expansion comment), not just date — same file, single edit.

Risks & Mitigations

• Trailing whitespace risk (per plugin-update: add Step 0 git-state preview + commit/push confirmation dialog before mutating origin #60 verify finding) — keep edits surgical; manual review of diff before commit.
• CHANGELOG cross-version ordering ([docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52) — verify [2.18.1] [2.18.0] [2.17.0] [2.16.1] [2.16.0] ordering after insert.
• changelog-validate 3-way sync — descriptions byte-equality check still applies; [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64 only changes CHANGELOG, not plugin.json/marketplace.json (those still have v1.16.0 description, which is different from v1.15.0). 3-way sync is on latest entry, not historical entries — this PR doesn't change latest, no sync issue.

Out-of-scope

• Parser hardening for [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 (Option B in issue) — defer
• Other CHANGELOG drift across marketplace plugins — defer to plugin-deploy audit pass
• README Workspace Patterns rewrite beyond [docs] README workspace pattern '100% backward compat' wording too strong (sister concern from #49) #55 — defer

[kiki830621]

Implementation Complete

Checklist (synced from cluster Implementation Plan)

• plugins/che-apple-mail-mcp/commands/archive-mail.md — Step 2.1 properties block, replace "Resilient to schema diversity" with v2.6+ format compatibility framing (Option A.1 per issue body) → commit 70e4485

Cluster context

Part of 4-issue docs cluster sharing PR #71. Cluster branch idd/cluster-docs-followups-49-60.

Changes

• 70e4485: docs(archive-mail): reduce 'schema diversity' claim to v2.6+ format compat (Refs [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54)

PR

#71

Verification

• ✅ Diff matches issue body's diff sketch (Option A.1 path)
• ✅ No code change — properties block claim wording only
• ✅ Future hardening reference kept honest (mentions yq parser / robust awk for follow-up)

Scope Compliance

Pure claim reduction. No parser hardening (Option B) — explicitly deferred per issue body until N≥3 user reports.

Sister Bugs Filed

none surfaced.

Next: verification pending

/idd-verify --pr 71

[kiki830621]

Verify (via PR #71)

Result: PASS — all blocking findings resolved in fix commit c04f66d
Full report: #71 (comment)

This issue's findings: see "#54" section in the linked report.

[kiki830621]

Closing Summary

Problem

archive-mail.md Step 2.1 sibling-archive dedup extension 的 properties block 宣稱 "Resilient to schema diversity" — 但實際 awk parser head -30 | awk '/^message_id:[ \t]*/{sub(/^message_id:[ \t]*"?/,"");sub(/"?$/,"");print;exit}' 只 handle archive-mail v2.6+ 自產的雙引號格式(message_id: "<...>")。對單引號 / inline comment / CRLF / trailing whitespace 等 manual-archive 變體會產生malformed key(非 silent skip),造成 dedup miss。

原文 README claim "100% backward compat" 跟 properties block "Resilient to schema diversity" 兩者都是 overpromise,被 /idd-verify PsychQuant/psychquant-claude-plugins#49 logic F5 + codex P3 + DA verification 抓到。

Root Cause

Parser 用 awk 寫的精簡 inline:sub(/^message_id:[ \t]*"?/,"") 與 sub(/"?$/,"") 中的 "? 是 optional,所以對單引號輸入 message_id: '<abc>',parser 不報錯,而是輸出 '<abc>'(包含單引號)。這個 string 進 in-memory dedup set。Step 4 比較 archive-mail canonical Message-ID <abc> 時對不上 → re-archive(dedup miss)。

Empirically verified during /idd-verify PR #71:bash awk test 印 '<abc>' 為 single-quoted input,confirms non-empty malformed-key behavior。

Solution

Option A.1 per issue body — claim reduction, not parser hardening:

1. Replace "Resilient to schema diversity" claim with "archive-mail v2.6+ format compatibility" framing
2. Explicit document failure mode is malformed-key dedup-miss, not silent skip(這是 verify-fix 的精準度修正)
3. Future hardening (yq parser / robust awk handling 單引號/inline comment/CRLF/trailing whitespace) deferred until N≥3 user reports of manual-archive missed dedup

Pure docs honesty fix — no code change to parser itself。

Verification

• 6-AI cluster verify on PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 returned PASS after Codex MEDIUM finding upheld and fixed in c04f66d(原本 verify-fix docs: Blog — Read Source Code at Runtime, Not Cache It #1 wording "silent skip → ENTRIES_THIS_DIR=0" 不準確,corrected 為 "malformed key, dedup miss")
• Empirical awk test confirms behavior:echo "message_id: '<abc>'" | awk ... outputs '<abc>'(non-empty,malformed)
• Cross-model dissent caught the accuracy issue:Claude reviewer flat-passed,Codex GPT-5.5 caught it
• Full verify report: PR PsychQuant/psychquant-claude-plugins#71 issuecomment-4415171847

Changes

• 70e4485 — docs(archive-mail): reduce 'schema diversity' claim to v2.6+ format compat (Refs [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54)
• c04f66d — fix(docs-cluster): apply /idd-verify PR cluster: 4 docs follow-ups from #49 / #60 IDD lifecycles #71 codex findings (Refs [docs] Backfill CHANGELOG [2.16.1] entry (sister concern from #49) #52 [docs] archive-mail YAML message_id parser brittle — reduce 'schema diversity' claim or strengthen (sister concern from #49) #54 [docs] plugin-tools CHANGELOG [1.15.0] entry has placeholder date — backfill (tangential from #60) #64) — corrected parser failure-mode description from "silent skip" to "malformed-key dedup-miss"
• 1557fbf — cluster squash merge

Cluster context

Part of 4-issue docs cluster (PR #71)。Sister issues #52 #55 #64 各自獨立 close。

Closing Follow-ups Filed (v2.45.0+ #527)

(none — no orphan mentions in closing summary)

Trigger-phrase scan hits: all classified as cross-linked context or CONSTRAINT-with-trigger-condition (IC_R011 §3 exempt) — no separate filings warranted.