How to Verify robots.txt rules

Overview

The robots.txt file guides crawler access. This check verifies the file exists, is syntactically valid, and does not block search engines or our crawler.

Severity Impact

A missing robots.txt file is a **Critical Site-Wide Issue** (-10 points).

Detection Logic

Fetches /robots.txt and parses line properties.

How to Fix

1. Create a valid robots.txt at your site root.
2. Ensure you allow crawler access (like Allow: /) for key search engines and our agent:

User-agent: CrawlSageBot
Allow: /

Related Guides

• Troubleshooting robots.txt Blockages