Skip to main content
Glossary

Crawl Error

By ยท Updated
Quick definition

Any HTTP failure (404, 500, timeout, blocked resource) a search engine hits while trying to fetch a URL. Reported in Google Search Console under Coverage.

Crawl Error in plain English

When Googlebot tries to fetch a URL and fails, the failure type is recorded. The main categories: 4xx errors (page not found, page gone, page forbidden), 5xx errors (server crashed, server overloaded, timeout), redirect issues (chains too deep, redirect loops), and resource-blocked errors (robots.txt disallowed, server rejected the request). Each type tells you something different about what's wrong.

4xx errors are usually content issues โ€” links to URLs that don't exist, deleted pages still in sitemaps, broken internal navigation. 5xx errors are infrastructure issues โ€” server is crashing, database queries are slow, the app is running out of memory. The two require completely different fixes. 5xx errors are particularly dangerous because Google interprets a pattern of 5xx responses as 'this site is unreliable, slow down crawling' โ€” which means your important pages get crawled less often.

Google Search Console's Coverage report (now called Indexing โ†’ Pages) groups crawl errors by type and shows trend data. A sudden spike in 5xx errors usually means a deploy broke something or traffic overwhelmed your server. A gradual rise in 404s usually means a CMS or sitemap is out of sync with reality. Either way, the spike is the signal.

The fix flow: triage by status code. For 4xx: are these URLs supposed to exist? If yes, fix the link or restore the page. If no, leave them โ€” Google will deindex. For 5xx: check server logs around the spike time, investigate the underlying infrastructure issue. For redirect chains: audit with Screaming Frog and collapse chains to single hops. For robots.txt blocks: review whether you intended to block those URLs.

Why crawl error matters for ecommerce

Crawl errors are the early-warning system for SEO disasters. A migration that broke URL handles, a CDN misconfiguration that made some pages serve 502s to Googlebot, a sitemap generator that's stale โ€” all of these surface in Search Console's crawl error reports before they cost you traffic. Set up email alerts for new errors and check the report at least weekly. For ecommerce stores especially, crawl errors during peak retail traffic (Black Friday week) directly translate to lost organic revenue.

Deeper dives on this term

Focused pages that go deeper than the definition โ€” comparisons, platform-specific guides, operational walkthroughs.

Compare

Crawl Error vs 301 Redirect: What's the Difference?

Crawl error vs 301 redirect: definitions, mechanics, and key differences every ecommerce operator needs to manage site health and

Read →
Compare

Crawl Error vs 404 Error: What's the Difference?

Crawl error vs 404 error: understand the key differences, how they overlap, and what each means for your ecommerce store's SEO hea

Read →
Compare

Crawl Error vs noindex: What's the Difference?

Crawl error vs noindex: understand the exact difference, how each affects Google's ability to find and show your pages, and which

Read →
Compare

Crawl Error vs robots.txt: What's the Difference?

Crawl errors vs robots.txt: understand how each blocks or breaks crawler access, how they differ, and when one masks the other.

Read →
Compare

Crawl Error vs Sitemap.xml: What's the Difference?

Crawl error vs sitemap.xml: understand what each term means, how they interact, and what ecommerce operators should prioritize fix

Read →
Platform

Crawl Error for Shopify Stores

Crawl errors on Shopify stores: platform-specific causes, Google Search Console signals, Liquid template limits, and actionable fi

Read →
Platform

Crawl Error for Wix Stores

Crawl errors on Wix stores explained: platform-specific causes, Wix SEO tools, app limits, and concrete fixes for ecommerce operat

Read →
Platform

Crawl Error for WooCommerce Stores

How crawl errors affect WooCommerce stores specifically โ€” platform conventions, common causes, WordPress plugins, and fixes for 6-

Read →
How-to

How to implement crawl error for an Ecommerce Store

Step-by-step guide to finding, diagnosing, and fixing crawl errors on an ecommerce store to protect organic rankings and indexatio

Read →
Checklist

Crawl Error Checklist: 12 Items Every Ecommerce Store Should Audit

A 12-item crawl error audit checklist for ecommerce stores. Each check includes a clear pass/fail criterion to fix indexing issues

Read →

Frequently asked questions

Where exactly do I find crawl errors in Google Search Console?

Indexing โ†’ Pages โ†’ scroll to the 'Why pages aren't indexed' section. Each error type (Not found 404, Server error 5xx, Excluded by 'noindex' tag, Page with redirect, etc.) is listed with affected URL counts. Click any error type to see the specific URLs and their last-crawled date. Bing Webmaster Tools has an equivalent at Reports & Data โ†’ Page Discovery.

How often does Google re-check pages with crawl errors?

Google retries 404s and 5xx errors on a decreasing schedule โ€” initially every few days, then weekly, then monthly. After about 30 days of consistent 404 responses, a URL is removed from the index. For 5xx errors, Google is more patient (assumes the site might recover) but if 5xx persists, crawl rate drops sharply across the whole site as a defensive measure.

Should I fix every crawl error?

No โ€” triage by impact. URLs that were never important (one-off typo'd inbound links, internal-only paths) can be ignored as long as your custom 404 page is helpful. Focus fixing on: (1) URLs with inbound links from external sites, (2) URLs that used to rank for important queries, (3) URLs in your XML sitemap that are 404ing (these should ALWAYS be fixed because the sitemap is your promise to Google about which URLs exist).

Why does Search Console show 'soft 404' for some pages that look fine to me?

Soft 404 means the page returned HTTP 200 but Google judged the content as not satisfying the URL's apparent intent โ€” usually because the page is mostly empty, has minimal text, has a 'page not found' message, or redirects to a homepage. Review the flagged URLs and either add substantial content (so it's a real page) or return the correct status code (404 for missing content).

Do crawl errors hurt rankings of other pages on my site?

Indirectly. A high-error site signals unreliability, which can throttle Google's overall crawl rate of your site โ€” important pages get re-crawled less often, so updates take longer to reflect in rankings. Severe 5xx error patterns can cause Google to temporarily back off crawling entirely. Healthy crawl error reports keep the rest of your site rankable.

MG
Written by

Matt is the founder of RunOctopus. He built All Angles Creatures from zero to page-1 rankings in reptile feeder insects in under 60 days using exactly this method โ€” turning a hard, entrenched niche into RunOctopus's proof store for programmatic SEO and AI search citation.

Connect on LinkedIn →

See what Otto would build for your store

Free architecture preview. No card required. Five minutes.

Generate Preview →