Audit AI crawler access and llms.txt configuration pre-migration.
- Audit AI crawler access and llms.txt configuration pre-migration2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
AI assistants require unrestricted crawler access and accurate llms.txt files to cite your site, crucial during domain migration to preserve AI visibility.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Review robots.txt for any disallow rules blocking known AI bot user agents like GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot.
- Confirm llms.txt exists at site root and includes up-to-date structured descriptions of your organization, services, and key page URLs.
- Verify no authentication or firewall rules inadvertently block AI bots by checking server logs during test crawls.
- Validate llms.txt format and content against official guidelines, ensuring concise, accurate, non-promotional information.
- Document findings and fix any crawler access issues and llms.txt errors before initiating domain migration.
Watch for
Blocking AI crawlers by robots.txt or having outdated llms.txt can prevent AI engines from crawling your content, losing citations and visibility.
Definition of done
Validated robots.txt allows all major AI assistant bots access; llms.txt is present at domain root, accurately describes key entities and URLs without blocking.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.