Say hello to CrawlSage AI — your autonomous SEO engineer. Start your first audit now.
CrawlSage
Documentation Menu↓

Installation

Get CrawlSage set up on your project in just a few steps. We've designed the onboarding process to be as frictionless as possible.

Requirements

Before you begin, ensure you have the following:

  • A CrawlSage account (you can sign up at app.crawlsage.com).
  • Access to the website you want to audit (admin access is preferred but not required for basic audits).
  • A modern web browser (Chrome, Firefox, Safari, or Edge).

1. Create your account

Visit our application portal and sign up with your email. You'll receive a verification link to activate your account. Once verified, you can start using your free daily credits immediately.

2. Configure your project

After logging in, click "New Project". You'll be prompted to enter your website's primary URL. CrawlSage will perform a quick pre-scan to detect your site's structure and CMS (WordPress, Next.js, etc.).

Pro Tip:

If you have a large site (10,000+ pages), we recommend starting with a Quick Scan to get immediate results while the full audit runs in the background.

3. Run your first audit

Select "Start Audit". Common modes include:

  • Full Audit: Broader crawl with SEO + technical checks.
  • Single Audit: Focused analysis of one page.
  • Performance Audit: Core Web Vitals and performance bottlenecks.

4. Connect WordPress (optional)

If you want to apply approved SEO changes directly, open the audit installation/integration settings and connect your WordPress credentials.

  • Use a WordPress user with permissions to edit target content.
  • Verify the connection status before applying fixes.
  • Preview changes first, then apply only approved updates.

Local development

For local setup, configure all three CrawlSage projects and their environment files before starting.

  1. Backend: copy crawlsage-backend/.env.example to .env and set database/auth/AI values.
  2. Dashboard app: copy crawlsage-app/.env.example to .env and point API URL to backend.
  3. Website: copy crawlsage-website/.env.example to .env and set NEXT_PUBLIC_APP_URL and NEXT_PUBLIC_SITE_URL.
# example website env
NEXT_PUBLIC_APP_URL=http://localhost:3000
NEXT_PUBLIC_SITE_URL=http://localhost:3001
NEXT_PUBLIC_GA_ID=

Troubleshooting

If CrawlSage is unable to reach your site, check the following:

  1. Robots.txt: Ensure your robots.txt file isn't blocking our crawler (User-Agent: CrawlSageBot).
  2. Firewall: If you use a WAF (Cloudflare, Sucuri), you may need to whitelist our IP range.
  3. SSL: Ensure your SSL certificate is valid and not expired.