[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-25

[github-actions[bot]]

Summary

• Date of test: 2026-06-25
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page immediately communicates the value proposition clearly, and the Quick Start guide has a solid step-by-step structure — but a beginner will stumble on the COPILOT_GITHUB_TOKEN concept and the unexplained jargon (frontmatter, lock file) before they can run their first workflow.

---

🔴 Critical Issues Found

1. COPILOT_GITHUB_TOKEN confusion — may block setup entirely

URL: /gh-aw/setup/quick-start/ → Step 2

The Quick Start tells users to set up COPILOT_GITHUB_TOKEN described as "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". For a brand-new user this is immediately confusing:

• Why do I need two different GitHub tokens?
• What makes this token "Copilot access"?
• The Note box says to set Copilot Requests to Read under Permissions → Account permissions — but where is this setting, and what type of PAT is it (fine-grained vs. classic)?

A beginner will likely get stuck here and abandon the setup. The current inline Note feels like a footnote rather than a prerequisite.

📎 [quick-start-token-note.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/df9531ff6d6ccbbe7e95db408110825d6fc020b2898fd3b405cbb402d3a640a9.png?raw=true

2. Broken PDF on home page (console error)

URL: /gh-aw/ (Home)

The browser console reports: Failed to load PDF slides — InvalidPDFException. Something on the home page is attempting to load a PDF and failing silently. While invisible in normal browsing, it indicates a broken asset that should be fixed.

---

🟡 Confusing Areas

3. add-wizard command name is cryptic

URL: /gh-aw/setup/quick-start/ → Step 2

The very first command a new user runs is gh aw add-wizard githubnext/agentics/daily-repo-status. A beginner will wonder: what is a "wizard"? Is this an AI wizard? A setup assistant? The command name is not intuitive and there's no one-line explanation of why it's called add-wizard vs just add.

📎 [quick-start-step2.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/0589de03a2ce5f163db82ada26757ab656ef28d15afd2d0bcfd5c6ab3239b91c.png?raw=true

4. Jargon introduced without beginner-friendly definitions

URL: /gh-aw/setup/quick-start/

Three technical terms appear without upfront explanation:

• Frontmatter — the first mention is deep in Step 4; a beginner won't know what this is. The link to /gh-aw/reference/frontmatter/ is helpful but the term itself needs a 1-sentence inline definition on first use.
• Lock file / .lock.yml — explained in Step 2 but buried mid-paragraph. A beginner might panic seeing a file they "must never edit" being added to their repo without a clear callout.
• Engine — used in ---engine: claude--- without explaining that this is the AI provider choice.

5. "Peli's Agent Factory" in the main navigation

URL: All pages (sidebar nav)

The navigation includes "Peli's Agent Factory" as a top-level nav item. To a new user, this reads as someone's personal project page rather than official documentation. It has no descriptive tooltip or subtitle. A new user won't know whether to click it or if it's relevant to getting started.

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b4e3576233b9f7ee229415272106d03f97f461e82f6c295080191540ea6586a0.png?raw=true

6. CLI Commands page is overwhelming for beginners

URL: /gh-aw/setup/cli/

The page starts with a useful command table, then immediately dives into: pinning versions, a standalone installer, GitHub Actions setup action, GitHub Enterprise Server support, GHE Cloud with data residency, and custom GHE configuration scripts — all before explaining the basic commands. For a first-time user, 90% of this page is irrelevant noise that obscures the 10% they actually need.

📎 [cli-commands.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b3376b6d8cac8af104b4173d4d7bf5158c5a295b284ace9248ffdb89a8022c45.png?raw=true

7. GitHub Actions enablement check is vague

URL: /gh-aw/setup/quick-start/ → Prerequisites

The prerequisite "GitHub Actions enabled — Check in Settings → Actions" doesn't tell users what to look for. What should the setting say? What's the default? What do they do if it's disabled? A screenshot or a link to GitHub's docs on enabling Actions would reduce confusion.

---

🟢 What Worked Well

1. "Estimated time: 10 minutes" at the top of Quick Start — immediately sets expectations.
2. Prerequisites section — clearly enumerated with version requirements (gh v2.0.0+) and specific verification commands (gh auth status).
3. "Most Common Commands" table at the top of CLI Commands page — the best part of the CLI page; exactly what a new user needs.
4. Multiple AI engine support — prerequisites clearly list Copilot, Claude, Codex, Gemini with direct links.
5. Alternative: Standalone Installer — great fallback for restricted environments.
6. add-wizard prompts for missing secrets — the fact that the wizard handles secret setup interactively is mentioned; this is a huge UX win that deserves more prominent placement.
7. "What's next?" section — links to examples and further learning at the end of Quick Start.

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b233480b58829ccc9ecf0fd7bb9425d8e77fca7e5c7f80391cdb4e654469f671.png?raw=true

---

Recommendations

Quick wins (high impact, low effort)

1. Promote COPILOT_GITHUB_TOKEN to its own prerequisite step — Add a dedicated sub-step before Step 2 with a numbered checklist and a direct link to the PAT creation page with exact permission settings. Consider renaming or abbreviating in docs to reduce cognitive load.
2. Add a one-line definition on first use for frontmatter — e.g. "the configuration block between the --- markers at the top of each workflow file" (currently only appears in Step 4).
3. Rename "Peli's Agent Factory" in navigation to something like "Workflow Examples" or "Community Workflows" and add a short subtitle.
4. Add a >beginner / >advanced toggle or callout on the CLI page to separate essential commands from GHE/enterprise content.

Medium-term improvements

5. Explain the add-wizard vs add distinction inline — a one-sentence tooltip like "add-wizard = interactive guided setup; add = non-interactive" would prevent confusion.
6. Add a "What is a lock file?" callout box in Step 2 with a visual diagram showing markdown → .lock.yml → GitHub Actions.
7. Fix the broken PDF asset on the home page.
8. Add a screenshot to the GitHub Actions enablement prerequisite showing the Settings → Actions page.

Longer-term documentation improvements

9. Add a dedicated "Key Concepts" page (or expand the glossary) covering: frontmatter, lock file, engine, safe outputs, sandbox — linked from the Quick Start.
10. Create a "5-minute video walkthrough" that mirrors the Quick Start steps — the video on the Quick Start page is a great start but it's not immediately obvious it demonstrates the exact steps on the page.

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b4e3576233b9f7ee229415272106d03f97f461e82f6c295080191540ea6586a0.png?raw=true
📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b233480b58829ccc9ecf0fd7bb9425d8e77fca7e5c7f80391cdb4e654469f671.png?raw=true
📎 [quick-start-step2.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/0589de03a2ce5f163db82ada26757ab656ef28d15afd2d0bcfd5c6ab3239b91c.png?raw=true
📎 [quick-start-token-note.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/df9531ff6d6ccbbe7e95db408110825d6fc020b2898fd3b405cbb402d3a640a9.png?raw=true
📎 [cli-commands.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b3376b6d8cac8af104b4173d4d7bf5158c5a295b284ace9248ffdb89a8022c45.png?raw=true

---

References: §28148121017

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 66.4 AIC · ⌖ 11.6 AIC · ⊞ 6.5K · ◷

• expires on Jun 25, 2026, 9:13 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-26T05:13:33.253Z.

Closed by Workflow