Audit robots.txt and llms.txt for AI crawler access on domain migration.
- Audit robots.txt and llms.txt for AI crawler access on domain migration2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
Ensuring AI crawlers have proper access via robots.txt and llms.txt after domain migration is critical for answer engine optimization (AEO) and maintaining AI citation visibility.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Retrieve and review the site's robots.txt post-migration for any disallow rules blocking AI crawlers like OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot.
- Publish or update the llms.txt file at the domain root with accurate descriptions of organization, products, and key URLs.
- Use AI crawler testing tools or log analysis to confirm robots.txt and llms.txt accessibility by AI bots.
- Coordinate with development and SEO teams to update robots.txt and llms.txt promptly if crawl blocking issues are identified.
- Document the changes and schedule periodic reviews of robots.txt and llms.txt especially after site updates or migrations.
Watch for
Blocking AI crawlers accidentally in robots.txt or omitting llms.txt can prevent AI assistants from crawling and citing your site, leading to visibility loss.
Definition of done
Robots.txt allows all priority AI crawlers, and a current llms.txt is published at the site root specifying key pages and organization info, verified accessible.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.