Skip to main content
SEO

Broken Links and Crawl Budget: What Googlebot Actually Wastes Time On

Your site has a crawl budget whether you use it or not. Googlebot spends a finite number of requests per day on your domain — small sites get hundreds, large ones tens of thousands. Every broken link you leave in a sitemap, nav menu, or template burns part of that budget on 404s and redirects instead of on pages you want ranked.

A single dead link is harmless. The damage comes from patterns: pagination pages with shuffled IDs, old promo URLs still linked in the footer, a renamed category that every product page still points to. Sites with thousands of broken internal links show measurably slower discovery of new pages — Google revisits known-dead URLs to check whether they came back instead of crawling fresh content. Fix the pattern, not the instance.

Also note: an internal link returns 200 even when it redirects to the right page. That's counted as pass. A 200 that hides a 301 chain isn't a working link; it's a crawl-budget tax — every hop burns budget and dilutes signals that could point at the final URL directly.

The four checks that matter, in order

Run them in this order and stop when one finds the problem; a full audit usually takes under an hour once you know where to look.

  1. Dead links (404). Run the broken link checker across the pages that matter — homepage, category hubs, top-converting pages. Ignore third-party links you can't control and focus on internal URLs. Each internal 404 is one you wrote, so it's one you can fix at the source in the template or database.
  2. Redirect chains. Use the URL redirect checker on the links you do keep. If a "working" link passes through two or three hops before landing, replace the href with the final destination. One hop for a deliberate migration is fine; chains are usually an accident nobody noticed because the browser followed them silently.
  3. Pages that are live but not indexed. A link that returns 200 and correctly redirects still doesn't help if Google never indexed the target. For key pages — money pages, new content — check actual inclusion with the Google index checker. If a page is live, linked, and still not in the index after a few weeks, the crawl budget is going somewhere else and your internal structure is part of the reason.
  4. Up-time on the pages you care about. Crawl budget also depends on how often Googlebot gets errors rather than pages. A flaky host returning intermittent 5xx makes Googlebot back off. A quick check of the pages you actually earn from — use the HTTP availability checker — tells you whether you have a server-reliability problem stacked on top of the link problem.

Where broken links actually come from

Almost never from "someone forgot." From real, repeatable causes:

  • Renamed slugs. You change one URL in the CMS, and forty internal links across related articles still point to the old address. The page itself 404s, and its former authority stops flowing.
  • Deleted pages that were linked. Pull a landing page for a finished campaign, leave the footer banner that goes to it. It sits there 404ing for months.
  • Stale sitemap. sitemap.xml still lists yesterday's test URLs, draft pages, or paginated archives you killed last quarter. Google reads a sitemap as an invitation — a bad one invites it to spend budget on junk.
  • Pagination with shuffled sort order. If "page 3" is ?sort=price&page=3 one day and ?sort=new&page=7 the next, crawlers see an infinite set of near-identical URLs and waste budget disambiguating them.
  • Relative-path bugs. A template hardcodes <a href="/old-category/item"> while the category was renamed. The link renders fine on the homepage, 404s from a subpage, and nobody catches it in testing.

Nobody on the team writes these on purpose. They accumulate because nobody had a cheap routine to look. The four checks above are that routine.

Fixing it, in practice

  1. Fix the source, not the symptom. When you find the same dead URL on multiple pages, fix it where it's generated — nav template, footer include, sitemap generator — not page by page.
  2. Redirect or remove, choose one. If a page is gone for good and has no successor, let it 404 — a deliberate, correct 404 wastes nothing. If it has a replacement, put a single 301 at the old URL pointing straight to it and update the links pointing there to the new address directly. Don't redirect to the homepage to "save" a URL; that's a soft 404 and Google treats it as junk.
  3. Clean the sitemap. Only URLs you want crawled belong in sitemap.xml. Run the broken-link checker against the sitemap as a separate pass; it's usually where the oldest, dumbest mistakes live because nobody reads it after the first launch.
  4. Re-check after every migration or rename. A redesign, platform swap, or category restructure creates broken links in bulk in one afternoon. The next scheduled check is weeks away — run the checker the same day instead of waiting.

Crawl budget you can spend elsewhere

Broken-link cleanup isn't glamorous, and the payoff isn't a single metric — it's fewer wasted requests and faster discovery of what you actually publish next. Sites that keep internal 404s near zero and chains at one hop get new pages crawled and indexed in days, not weeks. That speed compounds: publish more, iterate on titles and descriptions, run experiments on real pages while the site is still fresh in Googlebot's schedule.