robots.txt and AI crawlers
The problem
Many sites block AI crawlers without knowing it — sometimes a single Disallow: / under a wildcard rule does it.
What to check
Look for User-agent lines naming GPTBot, ClaudeBot, PerplexityBot, CCBot, or Google-Extended. A Disallow: / under any of those blocks that crawler entirely.
The fix
If you want AI systems to cite you, allow the crawlers you trust. Your report flags exactly which ones are blocked.