Audit and update domain migration robots.txt for AI crawler access.
- Audit and update domain migration robots.txt for AI crawler access2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
Ensuring AI assistants can access your new domain after migration is critical for maintaining and growing AI citation and visibility. Robots.txt misconfiguration often blocks AI crawlers unintentionally, hurting post-migration SEO.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Review your new domain's robots.txt file for disallow rules blocking AI crawler user agents
- Add explicit allow rules for known AI crawlers like GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot
- Test access for each AI crawler using live crawler user-agent simulations or logs
- Confirm key pages return successful HTTP status codes and crawlable HTML without JavaScript dependency
- Update and publish a current llms.txt file to guide AI crawlers on your important pages
Watch for
Blocking important AI crawlers in robots.txt will prevent citations and reduce AI visibility, harming brand presence in answer engines.
Definition of done
Robots.txt explicitly allows all major AI crawlers (e.g., GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot) without disallow rules on key content URLs, verified by direct crawl tests.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.