Per r/coolgithubprojects feedback from @scarletdawnredd: https://www.reddit.com/r/coolgithubprojects/comments/1tjjohw/comment/on2kjck/
The AI bot allowlist check (C-02) currently treats absence-of-explicit-allow as a meaningful failure signal. The critic correctly points out that robots.txt is opt-out, not opt-in. Absence of explicit allow doesn't block AI bots, they can still crawl.
The check is closer to "documentation hygiene" than "technical blocker."
Proposed change for v0.3.0:
- Move C-02 default severity from P1 to P2 (informational)
- Re-word the WARN message to "AI bots not explicitly named in robots.txt. They are not blocked, but you lose the documentation signal."
- Update CATEGORY_WEIGHTS so the check doesn't drag down Crawlability score as aggressively
- Calibration sites need re-validation after the change (work-smart.ai, stripe.com, example.com, WE Family)
Per r/coolgithubprojects feedback from @scarletdawnredd: https://www.reddit.com/r/coolgithubprojects/comments/1tjjohw/comment/on2kjck/
The AI bot allowlist check (C-02) currently treats absence-of-explicit-allow as a meaningful failure signal. The critic correctly points out that robots.txt is opt-out, not opt-in. Absence of explicit allow doesn't block AI bots, they can still crawl.
The check is closer to "documentation hygiene" than "technical blocker."
Proposed change for v0.3.0: