GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-05
Audit date: 2026-07-05
Run: §28747686600
Finding
The docs site (github.github.com/gh-aw/) has no robots.txt at all. The robots.txt check scored 0/18 — the single largest gap in the entire docs site audit. All 17 sitemap pages uniformly score in the Foundation or Critical band (32–44/100), and missing robots.txt (together with missing llms.txt) accounts for up to 36 of the lost points on every page.
From the audit JSON robots_txt.passed: false, robots_txt.details.found: false, and platform citation scores for ChatGPT (50/100) and Perplexity (65/100) both flag "Allow GPTBot in robots.txt" and "Allow PerplexityBot in robots.txt" as top recommendations.
Why this matters
Without a robots.txt, AI crawlers like GPTBot, ClaudeBot, and PerplexityBot cannot explicitly confirm crawl permission, causing them to be cautious about indexing the documentation site. This directly suppresses citation coverage across all major AI platforms (ChatGPT, Perplexity, Google AI Overviews). Fixing this single file unlocks +18 points on every page of the docs site.
Suggested fix
- Create
public/robots.txt in the docs site source (Astro project under docs/ or equivalent):
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
Sitemap: https://github.github.com/gh-aw/sitemap-index.xml
- Verify it is served at
https://github.github.com/gh-aw/robots.txt after deployment.
- Re-run
gh aw audit geo to confirm the robots.txt check jumps from 0/18 to 15+/18.
Generated by 🌍 GEO Optimizer Daily Audit · 37 AIC · ⌖ 17 AIC · ⊞ 5K · ◷
GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-05
Audit date: 2026-07-05
Run: §28747686600
Finding
From the audit JSON
robots_txt.passed: false,robots_txt.details.found: false, and platform citation scores for ChatGPT (50/100) and Perplexity (65/100) both flag "Allow GPTBot in robots.txt" and "Allow PerplexityBot in robots.txt" as top recommendations.Why this matters
Without a
robots.txt, AI crawlers like GPTBot, ClaudeBot, and PerplexityBot cannot explicitly confirm crawl permission, causing them to be cautious about indexing the documentation site. This directly suppresses citation coverage across all major AI platforms (ChatGPT, Perplexity, Google AI Overviews). Fixing this single file unlocks +18 points on every page of the docs site.Suggested fix
public/robots.txtin the docs site source (Astro project underdocs/or equivalent):https://github.github.com/gh-aw/robots.txtafter deployment.gh aw audit geoto confirm the robots.txt check jumps from 0/18 to 15+/18.