Best SEO Checklist
ONE FOCUSED ACTION

Audit and whitelist AI crawler access via robots.txt.

AI-search optimization · Answer engine optimization · Crawlability optimization · Domain migration SEO · Google AI Mode optimization · Google AI Overviews optimization · Indexability optimization · Knowledge-base SEO

1 checklist items

Save for later 0

Use × to move an item here. You can restore it when you’re ready.

    Your checks and saved items stay in this browser. Nothing is submitted.

    Why this matters

    AI assistants require explicit crawling permission to discover and cite your site content during domain migration.

    A STARTING-POINT EXAMPLE

    Hypothetical standard operating procedure

    Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.

    1. Review robots.txt for user-agent entries related to known AI crawlers (e.g., GPTBot, ClaudeBot, OAI-SearchBot).
    2. Remove or adjust disallow rules blocking these bots on key domains or pages.
    3. Test robots.txt changes with online tester tools to confirm bot access.
    4. Deploy updated robots.txt file to the new domain root.
    5. Monitor AI crawler logs or reports to verify successful access after migration.

    Watch for

    Blocking AI crawlers inadvertently can prevent your site from being referenced in AI-generated answers, harming visibility.

    Definition of done

    Robots.txt explicitly allows all relevant AI crawlers with no conflicting disallow rules on priority pages.

    Sources & attribution

    1. Answer Engine Optimization - AEO Toolkit ↗
    2. Website crawlability and discovery checklist for AI agents ↗
    3. Answer Engine Optimization in 2026: The Complete AEO Checklist - CiteMetrix ↗
    4. Technical SEO Checklist ↗
    5. SEO Technical Checklist ↗
    6. WordPress SEO Checklist for Publishers: On-Page and Technical Basics ↗
    7. Multilingual SEO checklist: What to verify before launch | SimpleLocalize ↗
    8. On-Page SEO Checklist for Small Businesses | BrandPepper ↗
    9. How to Optimize for Voice Search | Divramis SEO Agency ↗

    Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.