Understanding Code Generation Logic

Overview

The code generation pipeline converts technical SEO logs into structured HTML patches. This page details the sitemap parsing models, JSON extraction logic, and validation routines implemented in the AI engine.

How Code Generation Works

CrawlSage uses a multi-tier generation approach:

• DOM Extraction: The crawler downloads the target page and passes the metadata structures (current title, description, h1 count, and a text content snippet) to the AI service.
• Structured Prompting: The engine formats a prompt with exact schema structures and requests a JSON object from OpenAI model API.
• Sanitization & Parsing: The API output is stripped of markdown markers (e.g. ```json block tags) and parsed into variables.
• Fallback Handlers: If the model fails or outputs invalid schemas, robust fallback templates are triggered immediately.

Step-by-Step Guide

1. Trigger a website scan. The crawler checks all pages and flags issue codes.
2. For each issue (e.g. TITLE_LENGTH), the backend initiates a prompt building event.
3. The OpenAI service returns a signed suggestion object containing the value, reasoning, and confidence.
4. The suggestion is written to the database cache, rendering it immediately in your optimizations drawer.

Expected Result

You receive complete, syntax-validated tag corrections matching your page's semantic context, ready to be written to your database plugin.

Current Limitations

• **Context Limits:** The text content snippet sent to the API is capped at 800 characters to prevent prompt timeout.
• **OpenAI Cache:** Duplicate prompts use cached responses to save credits and decrease loading delays.

Frequently Asked Questions

• Does CrawlSage edit files directly?

  No. All fixes are applied via CMS plugin API calls that write to database meta fields.

Related Guides

• Evaluating AI Tag Recommendations
• Understanding the Deploy Pipeline