Allow OAI-SearchBot access in robots.txt.
- Allow OAI-SearchBot access in robots.txt1 cited source1 category
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
OAI-SearchBot crawling access is essential for ChatGPT to retrieve and cite your pages in real-time searches.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Review current robots.txt for OpenAI crawler rules.
- Add or confirm 'User-agent: OAI-SearchBot Allow: /' entry.
- Check CDN and firewall settings to ensure OAI-SearchBot is not blocked.
- Use AI crawler audit tools to verify accessibility.
- Wait 24 hours and re-test with OAI-SearchBot user-agent requests.
Watch for
Blocking OAI-SearchBot leads to exclusion from ChatGPT search citations, limiting AI visibility.
Definition of done
robots.txt includes User-agent: OAI-SearchBot Allow: /, verified via crawl diagnostics and AI crawler tests.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.