Verify robots.txt and llms.txt allow AI crawler access during domain migration.
- Verify robots.txt and llms.txt allow AI crawler access during domain migration2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
AI assistants rely on unrestricted crawl access and curated sitemap files to find citable pages. Blocking them hinders AI citations and visibility after a domain migration.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Review robots.txt to ensure no disallow rules block AI crawler user agents.
- Publish an up-to-date llms.txt at the domain root listing key pages and business info.
- Test site accessibility using AI crawler user-agent simulation tools or logs.
- Use crawler verification tools to confirm AI bots can fetch robots.txt and llms.txt without hindrance.
- Revalidate these settings immediately after migration launch and during ongoing audits.
Watch for
Blocking or misconfiguring robots.txt or omitting llms.txt can prevent AI crawlers from discovering your content, reducing AI-driven traffic substantially.
Definition of done
Robots.txt and llms.txt files explicitly allow major AI crawlers (e.g., OAI-SearchBot, GPTBot) access, verified by crawling tests and no crawler block logs.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.