Verify robots.txt and llms.txt allow AI crawler access pre-migration.
- Verify robots.txt and llms.txt allow AI crawler access pre-migration2 cited sources1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
Ensuring AI crawlers are permitted by robots.txt and that an accurate, curated llms.txt file exists at domain root helps AI assistants discover and cite your content after migration, avoiding accidental blocking that can reduce AI visibility.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Audit current robots.txt for disallow rules affecting AI crawler user agents like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot.
- Create or update llms.txt at the domain root describing your organization, products, and key pages clearly and in non-promotional language.
- Verify robots.txt entries explicitly permit AI crawlers mentioned by official docs (OpenAI, Anthropic).
- Confirm llms.txt content is current and reflects domain content and URLs, refreshing it after migration.
- Test access by fetching robots.txt and llms.txt as AI crawlers would, and use crawler emulation tools or logs to confirm crawl success.
Watch for
Blocking AI crawlers via robots.txt or having outdated llms.txt files may result in loss of AI citations and diminished answer engine presence.
Definition of done
Robots.txt explicitly allows major AI crawlers; llms.txt is present, concise, accurate, maps key authoritative URLs, and updates match site changes.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.