Best SEO Checklist
ONE FOCUSED ACTION

Verify robots.txt and XML sitemap entries for crawlability.

Crawlability optimization

1 checklist items

Save for later 0

Use × to move an item here. You can restore it when you’re ready.

    Your checks and saved items stay in this browser. Nothing is submitted.

    Why this matters

    Ensuring important pages are not blocked and sitemaps include only canonical, indexable URLs is crucial for optimal crawl efficiency and index coverage.

    A STARTING-POINT EXAMPLE

    Hypothetical standard operating procedure

    Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.

    1. Access your robots.txt file and confirm no important path or file is disallowed.
    2. Review your XML sitemap to ensure it includes only canonical URLs and excludes redirects or noindex pages.
    3. Use Google Search Console and Yandex Webmaster tools to analyze crawl errors and sitemap processing reports.
    4. Test robots.txt and sitemap URLs with search engine testing tools for correctness.
    5. Document findings and fix any discovered blocking or sitemap inclusion issues promptly.

    Watch for

    Avoid blocking essential pages in robots.txt or including non-canonical, redirected URLs in sitemaps that waste crawl budget.

    Definition of done

    Robots.txt allows crawler access to all priority pages and XML sitemap lists only canonical, indexable URLs without redirects or noindex pages.

    Sources & attribution

    1. SEO Technical Checklist - Rank Wiki ↗
    2. Yandex Webmaster - SEO Tools ↗

    Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.