How to Verify Live Fixes

Overview

CrawlSage incorporates an automated post-deployment validation engine. Whenever a code fix is deployed, our verification bot crawls the target URL to ensure the code changes are live on the page DOM. This prevents false positives and automatically handles rollbacks.

Prerequisites

• An applied recommendation in Processing or Applied state.
• A public website accessible to our crawler bot.

How Validation Works

1. Triggering Verification: The moment a code patch (such as a meta description update or JSON-LD schema block) is deployed via API, a verification task is automatically queued in our background workers (using BullMQ).
2. Cache-Busting Crawl: To bypass edge caching networks (like Cloudflare, Varnish, or local CMS page caches), the verification bot requests the page URL with an appended query parameter:

   ?crawlsage_nocache=1783969428

   This forces your origin server to generate and render a fresh HTML document.
3. DOM Inspection: The bot parses the returned HTML document's DOM structure. It looks for the unique CrawlSage tag identifier and checks that the inner tag properties match the approved values.
4. Status Compilation:
   • Success: If the tag matches, the recommendation state updates to Verified.
   • Failure: If the tag is missing or does not match, the verification task retries three times over 5 minutes. If all retries fail, a rollback is triggered.

Expected Result

Upon verification success, the recommendation card in your dashboard updates to a green Verified status badge. Your project's overall Health Score recalculates to award compliance points.

Best Practices

• Permit Query Parameters: Ensure that your server configurations do not drop or block custom query parameters. Stripping out the cache-busting timestamp will prevent our bot from bypassing edge caches.
• Verify Header Tags: Avoid hiding or deferring meta tags load using client-side JavaScript. The verification bot parses raw HTML tags, so tags should be rendered in the server-side markup.

Common Mistakes

• Blocking Scraper Bots: Whitelist the User-Agent: CrawlSageBot in your security dashboards to prevent the verification bot from being blocked.
• Overriding URLs: Changing target URLs during active validation runs will cause the verification task to fail. Wait for status markers to resolve before editing project scopes.

Troubleshooting

• Verification fails even though the tag is live on the site:
  1. Clear your local CMS cache (e.g. WP Rocket, W3 Total Cache) manually. The server might still be rendering cached pages to our bot.
  2. Inspect your security firewall logs to ensure requests containing query parameters like ?crawlsage_nocache= are not flagged as malicious scraper attempts.