Troubleshooting robots.txt Blockages

Problem

The crawler is blocked from indexing your pages, and all page audits report "Crawl Blocked by robots.txt".

Symptoms

• Audit reports display 0 crawled pages.
• The project health score drops to 0% with a "Robots Exclusion Active" warning.

Resolution

1. Open your site's `robots.txt` file (usually at https://yoursite.com/robots.txt).
2. Check for the rule:
   textCopy

   Disallow: /

3. Add an explicit exception rule for the CrawlSage crawler:
   textCopy

   User-agent: CrawlSageBot
   Allow: /