Verify that ChatGPT web crawler OAI-SearchBot is allowed in robots.txt.
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
OAI-SearchBot crawling permission is essential for real-time ChatGPT search citations and visibility.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Access your website's robots.txt file at yoursite.com/robots.txt.
- Search for 'User-agent: OAI-SearchBot' entry in the file.
- Ensure there is an 'Allow: /' directive under that user-agent to permit full crawling.
- Remove or adjust any 'Disallow' directives blocking OAI-SearchBot from important paths.
- Save and publish the updated robots.txt, then verify accessibility by testing with OpenAI crawler tools if available.
Watch for
Blocking OAI-SearchBot in robots.txt prevents your pages from being cited in ChatGPT web search results, reducing AI visibility.
Definition of done
Robots.txt explicitly allows OAI-SearchBot to crawl all important public content without disallow rules.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.