Audit and adjust robots.txt and llms.txt for AI crawler access during domain migration.
- Audit and adjust robots.txt and llms.txt for AI crawler access during domain migration2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
AI assistants rely on crawler access to index and cite your content; improper robots.txt or missing/incorrect llms.txt can prevent AI visibility after migration.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Review and remove disallow rules in robots.txt that block AI crawlers during migration.
- Publish an up-to-date llms.txt file at site root describing organization, key pages, and products.
- Test access by querying AI crawlers’ user agents or using crawler simulation tools.
- Validate llms.txt syntax and updated page links regularly, especially after migration.
- Monitor AI citations for expected pages post-launch to confirm access is working.
Watch for
Blocking legitimate AI crawlers or omitting key pages in llms.txt reduces chances of being cited and causes AI traffic loss.
Definition of done
Robots.txt allows all major AI crawlers (e.g., GPTBot, OAI-SearchBot) and llms.txt is published at domain root, accurately mapping important, citable pages.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.