Technical SEO Checklist: Crawling, Indexing, Sitemaps and robots.txt

Great content can’t rank if search engines can’t find it, read it, or store it. Technical SEO is the work that makes sure they can. Before a page earns a single impression, it has to pass through four stages:

1. Discovery: Google finds the URL through links, sitemaps, or redirects.
2. Crawling: Googlebot requests the page, subject to your robots.txt rules and server health.
3. Rendering and indexing: Google processes the page, including JavaScript, and decides whether to store it in its index.
4. Ranking: Only indexed pages can appear in search results.

A failure at any stage silently blocks everything after it. This checklist walks through the stages you control most directly: crawling, robots.txt, XML sitemaps, and indexing. Use it as a starting audit for a new site, a migration, or a site whose traffic has stalled.

1. Crawling checklist

Crawling is Googlebot fetching your URLs. Your goal is to make important pages easy to reach and cheap to fetch, and to stop wasting crawler time on pages that don’t matter.

• Keep important pages within 3 clicks of the homepage. Deeply buried pages get crawled less often and are treated as less important.
• Fix orphan pages. Every page you want indexed needs at least one internal link pointing to it. Compare a crawl (Screaming Frog, Sitebulb) against your sitemap to find pages with no inbound links.
• Use real <a href> links. Links built only with JavaScript click handlers or buttons may not be followed.
• Audit status codes. Important pages should return 200. Fix 404s that still receive links, and eliminate 5xx server errors, which cause Google to slow its crawling.
• Clean up redirects. Use 301 for permanent moves, point links directly at the final URL, and remove redirect chains and loops.
• Limit crawl traps. Faceted navigation, session IDs, infinite calendars, and internal search URLs can generate endless URL variations. Block or consolidate them.
• Watch server speed. Slow response times reduce how much Googlebot will crawl. Check the Crawl Stats report in Google Search Console for response time and error trends.
• Use HTTPS everywhere. Redirect HTTP to HTTPS and avoid mixed content.

A note on crawl budget

Crawl budget is the number of URLs Googlebot will fetch on your site in a given period. For most small and medium sites under roughly 10,000 URLs it is not a real constraint. It matters for large sites, such as ecommerce catalogs, publishers, and listing platforms, where thousands of low-value URLs can crowd out the ones that earn revenue.

2. robots.txt checklist

robots.txt is a plain text file at the root of your domain (https://example.com/robots.txt) that tells crawlers which paths they may request. It controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it.

• Place it at the root of each host. blog.example.com needs its own file, separate from example.com.
• Make sure it returns 200. A 5xx response can make Google pause crawling the site; a 404 is treated as “no restrictions.”
• Don’t block CSS, JavaScript, or images that pages need to render. Google must load them to see the page as users do.
• Don’t use it to hide pages from search. Use noindex for that, and let the page be crawled so Google can see the tag.
• Block low-value crawl paths: internal search results, cart and checkout steps, filter and sort parameters, and admin areas.
• Reference your sitemap with a Sitemap: line.
• Test changes with the robots.txt report in Search Console before deploying.
• Re-check after every launch or migration. A staging file with Disallow: / accidentally pushed to production is one of the most common and costly SEO mistakes.

Example robots.txt

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?sort=
Allow: /

Sitemap: https://example.com/sitemap.xml

Common mistakes

3. XML sitemap checklist

An XML sitemap is a list of the URLs you want search engines to find and index. It doesn’t guarantee indexing, but it speeds up discovery, especially for new, large, or poorly linked sites.

• Include only canonical, indexable URLs that return 200. Leave out redirects, 404s, noindex pages, and parameter duplicates.
• Respect the limits. A single sitemap can hold up to 50,000 URLs or 50 MB uncompressed. Beyond that, split into several files and list them in a sitemap index.
• Use absolute URLs with the correct protocol and hostname HTTPS, and with or without www consistently.
• Keep <lastmod> accurate. Update it only when the page content meaningfully changes. Google ignores <priority> and <changefreq>, so you can skip them.
• Submit it in Google Search Console and Bing Webmaster Tools, and reference it in robots.txt.
• Generate it automatically. Use your CMS or a plugin so the sitemap updates when content is added or removed.
• Split by content type posts, products, categories so you can spot indexing problems in each group using the Sitemaps report.
• Add specialized sitemaps if relevant, such as image, video, or news sitemaps.

Minimal sitemap example

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/technical-seo-checklist/</loc>
    <lastmod>2026-10-13</lastmod>
  </url>
</urlset>

4. Indexing checklist

A crawled page is not automatically indexed. Google decides whether a page is unique, useful, and technically sound enough to store. These checks help you control and diagnose that decision.

• Review the Page indexing report in Search Console. Look at the reasons pages are “Not indexed,” such as “Crawled currently not indexed,” “Discovered – currently not indexed,” and “Duplicate without user-selected canonical.”
• Use noindex deliberately. Apply <meta name="robots" content="noindex"> (or the X-Robots-Tag header for PDFs and files) to thin pages such as thank-you pages, internal search, and tag archives. Confirm it isn’t on pages you want ranked.
• Set a self-referencing canonical on every indexable page with <link rel="canonical" href="...">, and make sure it points to a live 200 URL.
• Consolidate duplicates. Handle www vs non-www, HTTP vs HTTPS, trailing slashes, and parameter variants with redirects or canonicals.
• Keep canonical signals consistent. The canonical tag, internal links, redirects, and sitemap should all point to the same preferred URL.
• Check JavaScript rendering. Use the URL Inspection tool’s “View crawled page” to confirm key content, links, and metadata appear in the rendered HTML.
• Avoid soft 404s. Empty or “not found” pages that return 200 confuse Google. Return a real 404 or 410.
• Improve thin or duplicate content. Pages that add little value are often crawled but left out of the index.
• Use the URL Inspection tool to test a live URL and request indexing for important new or updated pages.

Canonical vs. noindex vs. robots.txt

Quick-reference audit table

Use this table for a fast monthly check. Run through each row and note what you find.

Conclusion

Technical SEO is about removing friction. When crawlers can reach your pages, your robots.txt and sitemap give clear instructions, and your canonical and noindex signals agree with each other, your content gets a fair chance to rank. Start with the highest-impact checks: confirm robots.txt isn’t blocking anything important, submit a clean sitemap, and read the Page indexing report in Search Console. Then work through crawl efficiency and duplicate handling. Revisit the list after every redesign, migration, or major release, since that’s when most technical problems appear.