Best SEO Checklist
ONE FOCUSED ACTION

Audit and whitelist AI crawler access via robots.txt.

Domain migration SEO

1 checklist items

Save for later 0

Use × to move an item here. You can restore it when you’re ready.

    Your checks and saved items stay in this browser. Nothing is submitted.

    Why this matters

    AI assistants require explicit crawling permission to discover and cite your site content during domain migration.

    A STARTING-POINT EXAMPLE

    Hypothetical standard operating procedure

    Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.

    1. Review robots.txt for user-agent entries related to known AI crawlers (e.g., GPTBot, ClaudeBot, OAI-SearchBot).
    2. Remove or adjust disallow rules blocking these bots on key domains or pages.
    3. Test robots.txt changes with online tester tools to confirm bot access.
    4. Deploy updated robots.txt file to the new domain root.
    5. Monitor AI crawler logs or reports to verify successful access after migration.

    Watch for

    Blocking AI crawlers inadvertently can prevent your site from being referenced in AI-generated answers, harming visibility.

    Definition of done

    Robots.txt explicitly allows all relevant AI crawlers with no conflicting disallow rules on priority pages.

    Sources & attribution

    1. Answer Engine Optimization - AEO Toolkit ↗
    2. Website crawlability and discovery checklist for AI agents ↗

    Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.