From f4d5169403881aae6d011bb3e52a6d515f38238b Mon Sep 17 00:00:00 2001 From: farizanjum Date: Mon, 29 Jun 2026 21:04:42 +0530 Subject: [PATCH] docs: add AEO 1.1 draft spec --- spec/v1.1-draft.md | 390 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 390 insertions(+) create mode 100644 spec/v1.1-draft.md diff --git a/spec/v1.1-draft.md b/spec/v1.1-draft.md new file mode 100644 index 0000000..43572dc --- /dev/null +++ b/spec/v1.1-draft.md @@ -0,0 +1,390 @@ +# AEO Specification v1.1 Draft + +- **Status:** Draft for discussion (June 2026) +- **Extends:** [AEO Specification v1.0](./README.md) +- **Editors:** The Dualmark Working Group (Dodo Payments) + +This document proposes backward-compatible extensions to AEO Spec v1.0 for AI-oriented markdown discovery, section-level deep links, and structured extraction hints. AEO v1.1 aims to improve AI discovery, intra-document citation, and structured extraction while preserving all v1.0 conformance profiles. + +## 0. Status of this Document + +This document is a **draft proposal**, not a finalized conformance target. It has not been adopted by a standards body and is not yet required by the reference implementation. Its primary audience is implementors serving content to interactive assistants, AI search crawlers, offline indexers, and developer tools that retrieve markdown twins for citation or extraction. + +The keywords **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). + +## 1. Compatibility with AEO v1.0 + +A server conforming to [AEO Spec v1.0](./README.md) **MUST** remain conformant under this draft. The features in this document are additive extensions. + +Unless a future finalized v1.1 specification says otherwise: + +- Existing markdown twins do not need to change. +- Existing Basic, Standard, and Advanced conformance claims remain valid. +- Implementations **MAY** adopt `/sitemap.md`, section anchors, and structured-data hints independently. +- Clients **SHOULD** gracefully handle servers that implement only a subset of these features and **MUST NOT** assume that the presence of one v1.1 feature implies the others. +- Package implementations are expected to land in follow-up work after the draft receives feedback. + +## 2. Terminology + +- **Markdown Sitemap**: A markdown document, conventionally served at `/sitemap.md`, that lists canonical markdown twin URLs for AI agents. +- **Markdown Sitemap Entry**: One listed markdown twin in a Markdown Sitemap, optionally accompanied by metadata such as last-modified time and estimated token count. +- **Section Anchor**: A stable fragment identifier for an H2 or H3 section in a markdown twin, such as `#faq-refunds`. +- **Structured-Data Hint**: An opt-in fenced block in a markdown twin that mirrors schema.org JSON-LD or another machine-readable structured-data shape. +- **Canonical Markdown Twin URL**: The preferred `.md` URL for a page, following the v1.0 convention in [content-negotiation.md ยง6](./content-negotiation.md#6-canonical-markdown-url). + +## 3. Markdown Sitemap (`/sitemap.md`) + +The Markdown Sitemap is a markdown sibling to `sitemap.xml`. It gives AI agents a simple, text-first way to discover markdown twins without parsing XML or guessing URL patterns. + +A server **MAY** publish a Markdown Sitemap at `/sitemap.md`. If published, the Markdown Sitemap **SHOULD** list canonical markdown twin URLs rather than HTML URLs. + +### 3.1 Relationship to `sitemap.xml` and `llms.txt` + +The Markdown Sitemap does not replace `sitemap.xml` or `llms.txt`. + +- `sitemap.xml` remains the standard machine-readable sitemap format for crawlers. +- `llms.txt` is a curated AI-facing content guide, as described in [llms-txt-extensions.md](./llms-txt-extensions.md). +- `/sitemap.md` is a mechanical inventory of markdown twins. + +When both `llms.txt` and `/sitemap.md` exist, `llms.txt` **SHOULD** link to canonical HTML URLs and `/sitemap.md` **SHOULD** link to canonical markdown twin URLs. + +Publishers **SHOULD** generate `/sitemap.md` from the same route and freshness source of truth as `sitemap.xml` to avoid divergent URL or `last-modified` metadata. + +### 3.2 Format + +A Markdown Sitemap **SHOULD** begin with an H1 heading and then list markdown twin entries as markdown links. Headings are presentational; each bullet entry is the unit parsers should treat as one markdown twin. + +The minimal machine-oriented grammar is: + +1. A bullet line beginning with `- ` starts a new Markdown Sitemap Entry. +2. The bullet line **SHOULD** contain a markdown link whose target is the canonical markdown twin URL. +3. Indented lines, by at least two spaces or one tab, containing a `key: value` pair belong to that entry until the next bullet line or heading. Keys **SHOULD** use lowercase ASCII letters, digits, and hyphens. Values **MAY** be unquoted or quoted strings. +4. Blank lines between entries are allowed. Parsers **MUST** tolerate blank lines between metadata lines. + +Each entry **SHOULD** include: + +- A human-readable title. +- A canonical markdown twin URL. +- `last-modified`, as an ISO 8601 date (`YYYY-MM-DD`) or datetime (`YYYY-MM-DDTHH:mm:ssZ`). +- `tokens`, as a base-10 integer greater than or equal to `0`. + +Example: + +```markdown +# Sitemap + +- [Pricing](/pricing.md) + - last-modified: 2026-05-06 + - tokens: 842 +- [Refund FAQ](/faq/refunds.md) + - last-modified: 2026-05-01T12:30:00Z + - tokens: 391 +``` + +Publishers **MAY** group entries with H2 headings when helpful for readability: + +```markdown +# Sitemap + +## Product + +- [Pricing](/pricing.md) + - last-modified: 2026-05-06 + - tokens: 842 + +## Support + +- [Refund FAQ](/faq/refunds.md) + - last-modified: 2026-05-01T12:30:00Z + - tokens: 391 +``` + +Parsers **SHOULD** support both flat and grouped forms. Servers targeting minimal clients **SHOULD** prefer the flat form, while publishers **MAY** use grouping for human readability. + +### 3.3 Metadata semantics + +The `last-modified` value **SHOULD** describe the content freshness of the markdown twin, not merely the time the sitemap was generated. Clients **MAY** normalize datetimes to date-only values for indexing or cache bucketing. + +The `tokens` value **SHOULD** use the same estimation model as `X-Markdown-Tokens` where possible. Token counts are informational. Clients **MAY** use them for context budgeting. Clients **MUST NOT** rely on them for billing, authorization, or access control. + +If `tokens` is absent, clients **MUST** treat the token count as unknown, not as `0`. Clients **MAY** skip token-budget-based routing or fetch the markdown twin to read `X-Markdown-Tokens` when token metadata is missing. + +Each canonical HTML URL **SHOULD** have a single canonical markdown twin URL in `/sitemap.md`. Legacy or alternate markdown URLs **SHOULD NOT** be listed unless they are intentionally discoverable. Publishers **MAY** use an extension metadata key such as `canonical: true` while this draft settles canonicalization details. + +A Markdown Sitemap **MAY** contain absolute URLs, including URLs on language subdomains or alternate hosts, when those URLs are part of the same site inventory. For simpler cache and ownership semantics, each origin **SHOULD** publish its own `/sitemap.md` where practical. + +Parsers **SHOULD** ignore unknown metadata keys. Publishers **MAY** add extra keys in future drafts or implementation-specific extensions. + +### 3.4 Response headers + +A Markdown Sitemap response **SHOULD** use one of the following content types: + +- `text/markdown; charset=utf-8` +- `text/plain; charset=utf-8` + +A Markdown Sitemap response **SHOULD** set `X-Robots-Tag: noindex` and **MAY** set `Cache-Control: public, max-age=3600`. The `noindex` directive controls search-result visibility; it does not prevent AI agents or crawlers from fetching the sitemap for discovery. + +### 3.5 Shards and sitemap indexes + +Large sites **MAY** publish Markdown Sitemap shards such as `/sitemap-docs.md` and `/sitemap-blog.md`. If shards are used, `/sitemap.md` **SHOULD** act as the discoverable index that links to every shard. Clients **SHOULD NOT** be required to guess shard names. + +Example sitemap index: + +```markdown +# Sitemap + +## Sitemap Shards + +- [Documentation sitemap](/sitemap-docs.md) +- [Blog sitemap](/sitemap-blog.md) +``` + +Publishers **SHOULD** keep each shard small enough for practical parsing by AI agents. A future draft may define entry-count or token-count limits. + +## 4. Section Anchor Convention + +Section anchors let AI agents cite or retrieve a specific answer within a markdown twin: + +```txt +/pricing.md#faq-refunds +``` + +Markdown processors do not consistently generate heading IDs. AEO v1.1 therefore defines a stable convention for publishers and consumers. + +### 4.1 Anchor stability + +Publishers **SHOULD** provide stable anchors for H2 and H3 sections that are likely to be cited by AI agents, especially FAQ, pricing, comparison, policy, and documentation sections. + +Publishers **SHOULD** keep section anchors stable across content edits when the semantic meaning of the section has not changed. + +### 4.2 Explicit anchors + +For maximum compatibility, publishers **SHOULD** emit explicit HTML anchors immediately before the relevant heading: + +```markdown + + +### Can I get a refund? +``` + +CommonMark permits raw HTML, and explicit anchors avoid relying on renderer-specific heading slug algorithms. + +When an explicit anchor such as `` or `` is present immediately before or within a heading, consumers **MUST** prioritize that explicit identifier over any fallback slug. The fallback slug algorithm **MUST NOT** create a competing canonical anchor for a heading that already has an explicit anchor. + +Publishers **MAY** define multiple explicit anchors immediately before a heading to preserve legacy aliases: + +```markdown + + +### Can I get a refund? +``` + +Consumers **SHOULD** honor all explicit aliases. + +Publishers **SHOULD NOT** rely only on non-standard heading attribute syntax such as: + +```markdown +### Can I get a refund? {#faq-refunds} +``` + +unless they know their markdown renderer and AI consumers support that syntax. Consumers **MAY** parse and honor this syntax when their toolchain supports it, but explicit HTML anchors remain the portable form. + +### 4.3 Fallback slug algorithm + +When explicit anchors are not present, consumers **MAY** derive anchors from H2 and H3 text using the following fallback algorithm. Fallback anchors for H4, H5, and H6 headings are optional and implementation-defined. + +1. Strip inline markdown formatting. +2. Convert text to lowercase. +3. Trim leading and trailing whitespace. +4. Replace each run of non-alphanumeric characters with `-`. For this algorithm, alphanumeric characters **MUST** include Unicode letters and numbers. +5. Trim leading and trailing `-` characters. +6. If the result is empty, use `section-N`, where `N` is the 1-based index of the H2/H3 heading in document order. +7. Append `-2`, `-3`, and so on for duplicate anchors in document order. The first occurrence has no numeric suffix; the second occurrence receives `-2`. + +Examples: + +| Heading | Anchor | +| ------------------------- | --------------------- | +| `## Refunds` | `#refunds` | +| `### Can I get a refund?` | `#can-i-get-a-refund` | +| `## ๐Ÿš€ Getting Started` | `#getting-started` | +| `## ๐Ÿš€` | `#section-1` | +| First `## Pricing` | `#pricing` | +| Second `## Pricing` | `#pricing-2` | + +Publishers **SHOULD** prefer explicit ASCII anchors for maximum compatibility, especially when they want semantic prefixes such as `faq-`. Publishers **MAY** use percent-encoded or Unicode anchors when required by their content, but consumers are not required by this draft to normalize every Unicode slugging strategy. + +Fragment identifiers are case-sensitive for purposes of this draft. Publishers **SHOULD** emit lowercase anchors and **SHOULD** avoid very long anchors when a shorter stable identifier would communicate the same section. + +## 5. Structured-Data Hints + +Structured-data hints let markdown twins expose schema-like facts without requiring HTML `