Crawl Error vs noindex: The Core Distinction
A crawl error means Googlebot attempted to fetch a URL and failed — the server returned a 4xx or 5xx status code, the DNS lookup broke down, or the connection timed out. The page is unreachable. A noindex directive, by contrast, is a deliberate instruction embedded in a page's HTTP response headers or HTML meta tag that tells search engines not to include the page in their index. The page is reachable; it is simply excluded from search results on purpose.
The simplest way to draw the line: a crawl error is an infrastructure problem, and a noindex is an editorial decision. One prevents Google from reading the page at all. The other lets Google read the page but prohibits it from ranking or displaying that page to searchers. Conflating the two leads to misdiagnosis — chasing technical fixes when the real issue is a misapplied directive, or vice versa.
How Each Mechanism Works
When Googlebot hits a crawl error, it receives a response code that signals failure. A 404 tells the bot the resource does not exist. A 500 tells it the server broke. A DNS error means the domain cannot be resolved. In all these cases, Googlebot logs the failure, schedules a retry, and — if the error persists — eventually stops attempting that URL. No content is read. No signals are passed. The URL is essentially invisible to Google's indexing pipeline.
A noindex directive works entirely downstream of a successful crawl. Googlebot fetches the page, reads its content, follows its links, and only then encounters the instruction not to index. Google respects the directive and drops the URL from search results, but it still consumes crawl budget to fetch the page. The distinction matters operationally: noindex requires a successful HTTP 200 response to function. If you place a noindex tag on a page that also returns a 404, Google never reads the tag, so the directive has no effect.
When Each Situation Applies in an Ecommerce Context
Crawl errors appear most frequently after site migrations, when product URLs change without proper redirects, when faceted navigation generates thousands of broken parameter URLs, or when a server is under load during a traffic spike. They are unintentional in almost every case. The goal is always to eliminate them — either by restoring the resource, setting up a 301 redirect to a canonical equivalent, or confirming the URL is legitimately gone.
noindex is intentional and appropriate in specific scenarios: staging environments, thank-you pages, internal search results, paginated collection pages beyond page two or three, and thin product variants that add no unique content. On a large catalog site, the noindex directive is an active editorial tool for managing which subset of URLs competes in search. The error is applying it accidentally — for instance, a developer leaving a sitewide noindex tag in production after a launch, a common mistake that can quietly suppress an entire store's organic visibility.
How Crawl Errors and noindex Interact
The two conditions can coexist, and understanding their interaction prevents diagnostic confusion. A URL can return a 200 status with a noindex tag — crawlable but not indexable. A URL can return a 404 — neither crawlable nor indexable, but for a completely different reason. A URL can also return a 200 without a noindex tag and still not rank, due to thin content or a lack of links. These are distinct problems requiring distinct solutions.
One interaction to watch: if a page is blocked by robots.txt and also carries a noindex tag, Google cannot read the noindex instruction because it cannot crawl the page at all. The robots.txt block wins. This means a page blocked in robots.txt is not reliably de-indexed; remnants of it can persist in Google's index from historical crawls. To remove a page from the index reliably, the URL must be crawlable (200 status, not blocked by robots.txt) and carry a noindex directive, or it must be removed via Google Search Console's URL Removal tool.
Diagnosing Which Problem You Actually Have
In Google Search Console, crawl errors surface in the Pages report under reasons like 'Not found (404)', 'Server error (5xx)', or 'Redirect error'. noindex issues surface under 'Excluded by noindex tag' or 'Excluded by 'noindex' (in the HTTP header)'. These are in separate buckets. A URL appearing in the noindex exclusion bucket is not broken — it is functioning as configured, intentionally or not.
The diagnostic sequence for any URL that should rank but does not: first confirm it returns a 200 status (no crawl error), then confirm it is not blocked by robots.txt, then confirm it carries no noindex tag. Run this check with the URL Inspection tool in Search Console, which reports the live HTTP status, robots.txt block status, and any noindex signals simultaneously. Fixing a crawl error on a page that also carries a noindex tag will not restore rankings — both conditions must be clear.
Actionable Priorities for Store Operators
Fix crawl errors that affect pages with commercial value first — category pages, top-selling product pages, and collection landing pages. A 404 on a high-revenue URL costs ranking authority immediately. Set up 301 redirects to the nearest relevant live page to preserve link equity. Schedule a monthly crawl error review in Search Console so new errors surface before they accumulate at scale.
Audit noindex usage quarterly. Export the full list of noindex-excluded URLs from Search Console and verify each exclusion is deliberate. Pay particular attention after platform updates, theme changes, or app installations, any of which can inject unexpected noindex tags. The goal is a clean separation: crawl errors at zero, noindex applied only to URLs that genuinely should not rank.