How to fix crawl errors in Google Search Console

To fix crawl errors in Google Search Console, open Page indexing and URL Inspection, then repair fetch problems in order: server errors and redirect chains, true 404s, soft 404s, and robots.txt blocks. Only after Googlebot gets a clean HTTP response should you tune content, canonicals, or sitemap entries. This guide maps the common GSC statuses to the smallest fix and shows how to verify the URL before you request indexing again.

Where crawl errors appear in GSC

Google Search Console splits "can Google reach this URL?" from "should Google store it in the index?" The Page indexing report is the daily starting point. It groups URLs by status, such as indexed, excluded, or error, and gives a reason label for each bucket.

URL Inspection is the single-URL debugger. It shows the last crawl time, the HTTP result, whether indexing is allowed, and the difference between crawled HTML and rendered HTML when JavaScript changes the page. Use the report for volume and trends; use inspection before you declare a fix done.

Crawl stats under Settings and legacy crawl-error views still matter for spikes in 5xx responses or sudden drop-offs in crawl requests. When Page indexing shows new "Server error" or "Redirect error" rows, treat that as a site-wide incident, not a content task.

For the wider checklist on discovery, sitemap submission, and first-time visibility, read why your site is not showing on Google. This page stays focused on errors that block or distort crawling.

The triage order that saves time

Teams waste weeks rewriting titles while URLs still return 500 or loop through redirects. Use this order so each fix unlocks the next layer of data.

1. Server error (5xx). Nothing else is trustworthy until origin responses are stable.
2. Redirect error. Fix loops, missing destinations, and http/https or www mismatches.
3. Not found (404). Remove dead links, update the sitemap, or add a purposeful redirect.
4. Soft 404. Stop returning 200 for empty or "missing" templates.
5. Blocked by robots.txt. Allow public paths; keep staging and admin disallowed.
6. Indexing exclusions with a working fetch. noindex, duplicate without user-selected canonical, crawled but not indexed.

Google's troubleshooting crawling guide follows the same principle: confirm availability, then review access and signals. Skipping the order produces false "fixes" that disappear on the next crawl.

Fix not found (404) and soft 404

A 404 Not Found in GSC means Google requested a URL your server does not serve. Common causes include deleted blog posts still linked from navigation, renamed routes without redirects, and sitemap generators that list old slugs.

Pick one response per URL:

• Restore or replace the content when the URL should exist.
• Return 404 or 410 when the resource is gone for good, then remove internal links and sitemap entries pointing at it.
• Issue a single 301 redirect when a clear successor URL exists. Avoid chaining through three hops.

A soft 404 is subtler: the status code is 200, but the body looks like a blank template, a generic "Oops" page, or a search results page with no results. Google treats that as "this is not a real document." Fix it by adding substantive content or by returning an honest 404. SPAs that render an empty shell until client JavaScript runs often trigger soft 404 until server rendering or prerendering exposes the main text in the first HTML response.

After you change routing, audit internal links so crawlers are not sent to dead paths. The guide on internal linking in HTML explains how to find orphan and broken targets in a crawl export.

Fix blocked by robots and server errors

Blocked by robots.txt means Googlebot was not allowed to fetch the URL. Robots.txt controls crawling, not indexing. A URL blocked from crawl may still appear in results if Google learned about it elsewhere, and blocking crawl also prevents Google from seeing a noindex tag on the page. Treat robots.txt as a coarse gate for paths, not a substitute for noindex on private pages.

Audit /robots.txt after every deploy that touches routes, CDN rules, or environment-specific hosts. Use the free robots.txt builder to draft rules, then validate them in Search Console's robots tester. Allow CSS and JavaScript your public pages need to render; accidental Disallow: /assets/ rules have blocked rendering on real sites.

Server error (5xx) rows mean Googlebot received an error from your origin or edge. Check application logs, rate limits, and deployment health first. Intermittent 502/503 responses during deploys often show up here as a crawl cliff. Fix infrastructure before you touch on-page SEO.

When errors cluster on one template, compare a failing URL in URL Inspection with the same path fetched through curl -I from outside your network. A WAF rule or geo block that spares your browser can still fail Googlebot.

Fix redirect errors and chains

Redirect error covers loops, too many hops, mixed protocols, and destinations that themselves error. Google documents redirect best practices in its 301 redirect guidance: one hop to the final URL, consistent HTTPS, and a 200 response at the end of the chain.

Typical fixes for SaaS marketing sites:

• Collapse http → www → https → trailing-slash chains into one 301 at the edge.
• Remove redirect rules that send old campaign URLs to the homepage when a specific replacement page exists.
• Ensure parameterized URLs either canonicalize to a clean version or redirect deliberately, not accidentally through middleware.

After you simplify redirects, re-run URL Inspection on the old and new URLs. The "Page fetch" section should show a single redirect and a successful final URL.

Verify fixes before you resubmit

Search Console's Request indexing button is a nudge, not a repair tool. Use it only after the underlying problem is fixed and verified.

A practical verification loop:

1. Fetch the URL with an HTTP client and confirm the expected status code and Location header on redirects.
2. Open URL Inspection, test the live URL, and read "Page availability" and "Indexing allowed?"
3. Compare crawled vs rendered HTML when the site relies on JavaScript for main content or internal links.
4. Request indexing once for a single important URL, or refresh the sitemap when many URLs changed together.

If the URL stays excluded after two genuine fixes and recrawls, stop hammering the button. Re-read the reason label, check duplication and internal links, and confirm the page adds something distinct. Repeated requests without a change waste time and hide the real issue.

When sitemaps and canonicals matter

Sitemaps and canonical tags do not fix broken HTTP responses or robots blocks. They matter after Googlebot can fetch the page.

Sitemaps help discovery and give Google a list of canonical URLs you care about. Submit yours under Indexing → Sitemaps, then compare discovered counts with the routes your app considers public. A sitemap full of 404s trains the wrong signal. Regenerate it from the same source that builds routes, include only indexable URLs, and set lastmod from real content or deploy timestamps. Google's sitemap build guide describes the contract.

Canonical tags tell Google which URL is primary when similar versions exist, such as tracking parameters, print views, or HTTP/HTTPS duplicates. Emit a self-referencing canonical on each main page and point variants at the clean URL. Keep non-canonical variants out of the sitemap. When GSC shows "Duplicate without user-selected canonical," pick one URL, implement rel=canonical, and align internal links.

Developer checklists for robots, sitemaps, and CI guards live in the SEO workflow for developers. Use Orchory free SEO tools for lightweight checks, but treat the rendered HTML and response headers as the source of truth.

Keep pages indexable after you ship

Crawl errors often return when new routes ship without updated robots rules, sitemap entries, or internal links. The durable pattern is to treat indexability as part of the page definition: status code, robots meta, canonical, crawlable navigation, and honest content in the first HTML response.

Orchory helps teams ship cite-ready pages that stay indexable: answer-first structure, supportable claims, and templates that do not accidentally noindex production routes. Research and prioritization stay in the product; your team keeps control of what goes live.

When you want that loop on your site, send your site to Orchory. Your first page is free and set up with you on a call. For the full research-to-page workflow, see the auto SEO tool overview. If exclusions look like competing URLs rather than broken fetches, read what keyword cannibalization is before you merge or split pages.

FAQs