Confirm site crawlability by checking robots.txt and sitemap inclusion.
- Confirm site crawlability by checking robots.txt and sitemap inclusion2 cited sources2 categories
Save for later 0
Use × to move an item here. You can restore it when you’re ready.
Your checks and saved items stay in this browser. Nothing is submitted.
Why this matters
If search engines cannot crawl your category pages, they won't appear in DuckDuckGo search results, limiting visibility.
Hypothetical standard operating procedure
Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.
- Retrieve and review robots.txt for disallow rules affecting category pages.
- Verify category URLs are present in XML sitemap files.
- Test sample category URLs for HTTP 200 status and crawl accessibility.
- Ensure no noindex tags are present on category pages.
- Confirm sitemap URLs match canonical category URLs.
Watch for
Do not accidentally block important category pages via robots.txt or noindex directives.
Definition of done
All key category URLs return HTTP 200 and are included in sitemap without being blocked or noindexed.
Sources & attribution
Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.