[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-22

[github-actions[bot]]

Summary

• Date of test: 2026-06-22
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page gives a solid first impression with a clear value proposition and compelling security story, but the Quick Start has friction points — especially around token setup, unexplained jargon, and a quirky nav item — that would slow or confuse a complete newcomer.

---

🔴 Critical Issues Found

1. Search is non-functional — shows a developer warning

When a new user clicks the Search button (Ctrl+K), they see:

"Search is only available in production builds. Try building and previewing the site to test it out locally."

This is a dev-server message that should never be visible to docs users. End-users don't know what a "production build" is; they just see a broken search box. Since search is often the first thing people reach for in large docs sites, this is a blocking problem.

📎 [search-broken.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2c0c54965a450542fe6be1dec76b518309427a8a498248f9522249e72b628165.png?raw=true

2. "Peli's Agent Factory" in the primary navigation bar

The top navigation bar includes: Quick Start | Create | Examples | Docs | FAQ | Blog | **Peli's Agent Factory**

As a new user, I have absolutely no idea who "Peli" is, or why this personal-branded link is in the main navigation between "Blog" and the GitHub icon. It looks like a broken or test link, and could make the product look less professional. If this is a showcase of community workflows, it deserves a more descriptive label like "Showcase" or "Community Workflows".

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3a39b16f99740fe848b655df45853debab8d47d00d8451a52e51c0293a1d292c.png?raw=true

---

🟡 Confusing Areas

1. COPILOT_GITHUB_TOKEN — why do I need two GitHub tokens?

Quick Start Step 2 says I need to set up COPILOT_GITHUB_TOKEN, described as:

"a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN"

A complete beginner's reaction: "Wait, I'm already logged into GitHub. Why do I need ANOTHER GitHub token? What's the difference between COPILOT_GITHUB_TOKEN and GITHUB_TOKEN?" The distinction is not explained. Even the NOTE box only says how to create it, not why a separate token is architecturally required. A one-sentence explanation ("The workflow runs as the default Actions token, which cannot make Copilot API calls — so a separate user-level token with Copilot access is required") would help significantly.

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5e8f47d82b963bc8eef0110314434809316f2c4bc74940b08af51ef264334299.png?raw=true

2. No recommendation on which AI engine to start with

The prerequisites list 4 AI options: Copilot, Claude, Codex, Gemini — with no guidance on which is easiest or cheapest for a first-timer. Most beginners will be stuck at this decision. A simple "If you have GitHub Copilot, start there — it's the simplest to set up" would unblock many users immediately.

3. "frontmatter" used without definition

In Step 4 (Customize), the guide says:

"If you changed the frontmatter (the configuration block between the --- markers...)..."

The parenthetical helps, but "frontmatter" appears multiple times in the docs without ever being explicitly introduced as a term. A new user who doesn't have a static site/YAML background won't recognize it. The Glossary page exists but isn't linked here.

4. "10 minutes" estimate is optimistic

The Quick Start says "Estimated time: 10 minutes." But for a complete beginner who has never used fine-grained PATs, this is closer to 25–30 minutes — navigating PAT permissions, understanding the Copilot Requests permission, adding a repo secret, etc. An inaccurate time estimate sets users up for frustration when they're still on Step 2 at minute 15.

5. gh aw add-wizard format not explained inline

The command gh aw add-wizard githubnext/agentics/daily-repo-status uses an (owner)/(repo)/(workflow-name) format. This is mentioned in the next line, but only after the code block. Beginners often copy-paste the command without reading the explanation — they'll wonder what "githubnext/agentics/daily-repo-status" means and whether to replace any of it.

6. Overwhelming sidebar for first-time visitors

The left sidebar has 100+ links across dozens of categories (Design Patterns, Practices, Specs, Troubleshooting, Experimental...). While comprehensive, it's intimidating for someone who just wants to get started. There's no visual hierarchy or "Start here" callout to guide new users to the 3–4 pages that matter most initially.

📎 [cli-full.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e01cb913978f2e9474fea931a78487a55731321cfc91b1447f315e7dd473db92.png?raw=true

7. "What's next?" links to "Peli's Agent Factory" again without context

The quick start ends with: "Explore some of these in Peli's Agent Factory." — linking to a blog post. A first-time user just finished their first setup; they want a logical next step, not a personal blog link they don't understand.

8. A 404 page was reachable from sidebar navigation

The URL /gh-aw/reference/authentication/ (which a user might guess from the sidebar label "Authentication") returns a 404 — the actual URL is /gh-aw/reference/auth/. While the Quick Start itself links correctly to /reference/auth/, the mismatch between label ("Authentication") and URL slug (auth) could trip up users bookmarking or sharing links.

📎 [404-auth-broken.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/51f5bf8ef3266df8ef6a66b601b8d752c5352b741b47cca8fa3680444cef75e8.png?raw=true

---

🟢 What Worked Well

• Clear value proposition on the home page — "Intelligent automation for GitHub. Run the coding agents you know and love, with strong guardrails and cost controls, in GitHub Actions." Immediately clear.
• Security architecture section — The layered defense diagram and explanations are excellent. This builds trust with technically-minded users.
• Prerequisites are explicit — Listing gh version, gh auth status, OS requirements, and action items (with links) for each is genuinely helpful.
• "Most Common Commands" table on CLI page — This is the best part of the CLI page. It immediately answers "what do I actually need?" without forcing users to scan hundreds of lines.
• add-wizard interactive flow — Having the tool check prerequisites, select an AI engine, and set up secrets automatically is a great UX choice. It's the right way to onboard new users.
• TIP and NOTE callout boxes — Used well in the Quick Start to handle authentication edge cases without cluttering the main flow.
• Video in Quick Start — A video showing the install-and-run flow is a great addition. It loaded successfully.

---

Recommendations

Quick wins (high impact, low effort)

1. Rename "Peli's Agent Factory" in the nav to something like "Community" or "Showcase" — or move it out of the primary nav entirely.
2. Add a one-sentence "why" to the COPILOT_GITHUB_TOKEN note explaining why a separate token is needed.
3. Add an AI engine recommendation at the prerequisites: "New to this? Start with GitHub Copilot — setup is simplest if you already have a Copilot subscription."
4. Revise the time estimate to "15–30 minutes (first time)" or split it: "5 min if you have your API key ready; 20 min if you need to create one."
5. Link "frontmatter" to the Glossary on first use in the Quick Start.

Medium-term improvements

6. Add a "Getting Started" section to the sidebar that highlights just 4–5 pages for new users, visually separated from the reference docs.
7. Add a redirect from /reference/authentication/ → /reference/auth/ to prevent 404s from guessed URLs.
8. Explain the (owner)/(repo)/(workflow-name) format before or inline with the add-wizard command, not after the code block.
9. Consider a "Choose your AI engine" page — a simple decision guide to pick between Copilot/Claude/Codex/Gemini based on existing subscriptions and cost sensitivity.

Longer-term

10. Progressive disclosure in the sidebar — collapse the reference sections by default; expand only "Setup" and "Guides" for first-time visitors using localStorage.
11. "What's next?" redesign — Replace the "Peli's Agent Factory" link with a curated 3-card grid: e.g., "More workflow examples," "Customize your workflow," "Understand the security model."

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3a39b16f99740fe848b655df45853debab8d47d00d8451a52e51c0293a1d292c.png?raw=true

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5e8f47d82b963bc8eef0110314434809316f2c4bc74940b08af51ef264334299.png?raw=true

📎 [cli-full.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e01cb913978f2e9474fea931a78487a55731321cfc91b1447f315e7dd473db92.png?raw=true

📎 [search-broken.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2c0c54965a450542fe6be1dec76b518309427a8a498248f9522249e72b628165.png?raw=true

📎 [404-auth-broken.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/51f5bf8ef3266df8ef6a66b601b8d752c5352b741b47cca8fa3680444cef75e8.png?raw=true

---

References: §27931614973

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 79.8 AIC · ⌖ 17.7 AIC · ⊞ 6.5K · ◷

• expires on Jun 22, 2026, 9:39 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-23T05:39:55.130Z.

Closed by Workflow