Best SEO Checklist
ONE FOCUSED ACTION

Audit robots.txt for blocking unintentional URLs affecting crawlability.

Crawlability optimization

1 checklist items

Save for later 0

Use × to move an item here. You can restore it when you’re ready.

    Your checks and saved items stay in this browser. Nothing is submitted.

    Why this matters

    Incorrect robots.txt rules can block important pages from crawling, wasting crawl budget and causing visibility loss.

    A STARTING-POINT EXAMPLE

    Hypothetical standard operating procedure

    Adapt this example to your site, access, tools, and change process. It is our suggested procedure, informed by the sources below.

    1. Retrieve and review the current robots.txt file from the site root.
    2. Identify all Disallow directives, focusing on broad rules and parameter patterns.
    3. Cross-reference disallowed URLs with XML sitemap and analytics to spot important pages.
    4. Ensure important pages are not blocked; faceted navigation parameters are either blocked or canonicalized properly.
    5. Document findings and recommend adjustments to allow crawl access to priority content.

    Watch for

    Blocking pages in robots.txt prevents crawling, making noindex tags ineffective and can lead to unindexed important content.

    Definition of done

    Robots.txt does not block any important or indexable URLs, including faceted navigation with search demand.

    Sources & attribution

    1. The Technical SEO Checklist: 30 Checks That Actually Matter ↗
    2. SEO Technical Checklist: What It Is and How It Works ↗

    Original checklist wording and example SOP by Best SEO Checklist, using source-grounded AI assistance. Editorial contact: Ted Kubaitis, @tedkubaitis on Teams. Check current source guidance before implementation.