How to Configure Your First Project

Overview

To audit website pages and apply SEO adjustments, you must create a project slot. This links your dashboard account to the targeted domain, sets boundaries for the crawler bots, and enables ownership verification keys.

Prerequisites

• An active CrawlSage account.
• The primary canonical URL of the website you wish to monitor.
• Access to edit target domain DNS registers or upload root files (only required if configuring automated deployment).

Step-by-Step Guide

1. Open the Websites Panel: Log in to the CrawlSage dashboard. Select the "Websites" tab in the left sidebar navigation panel, or click the primary "New Project" button on the welcome card.
2. Enter Your Site Details: Click the "Add Website" button. In the sidebar modal, configure the properties:
   • Website Name: A custom label to identify the project slot (e.g., My Store).
   • Target URL: The primary canonical web address (e.g., https://mybusiness.com). Include the secure protocol (https://).
3. Define Crawler Constraints: Scroll to settings options:
   • Crawl Page Limit: Set the maximum page crawl threshold (e.g., 500 pages for initial runs).
   • Monitoring Frequency: Toggle automatic weekly scan preferences.
4. Save Project: Click "Create Project". The dashboard will register the domain slot and display the verification panel.

Expected Result

Your site will appear in the Websites list view. A unique **CrawlSage Connection Key** will be generated. You can now run ad-hoc audits or pair integration connectors to apply code corrections.

Best Practices

• Verify Canonical Protocol: Test your homepage URL in a private tab first to ensure the entered address maps exactly to the live protocol.
• Whitelisting Scraper Bot: If your web hosting uses Cloudflare, Sucuri, or a custom firewall, verify that the firewall rules permit requests from User-Agent: CrawlSageBot.

Common Mistakes

• Registering local setups: CrawlSage bots crawl public web addresses. Registering addresses like localhost:3000 or local staging domains behind firewalls will fail.
• Entering multiple subdomains in one slot: Each project slot manages a single subdomain hierarchy. To audit both blog.site.com and store.site.com, create separate projects.

Troubleshooting

• Verification fails or target is offline:
  1. Verify that your site is public and accessible in standard browsers.
  2. Check your robots.txt file to ensure it doesn't block crawler bots. See the robots.txt Troubleshooting manual.