Broken internal links: helping crawlers find every page

Broken internal links stop crawlers reaching pages, and they are only one of several link problems that hide content. Crawlers find most pages by following <a href> links from pages they already know, so a page with no working link to it, or one reachable only through a JavaScript click handler, may never be fetched at all. The fix is plain HTML: real links, pointing straight at live URLs, from pages that are themselves easy to reach.

This guide covers how link discovery works, the common ways it breaks (orphan pages, JavaScript-only navigation, dead links and redirect hops) and the commands to find each one.

How crawlers discover pages through links

Google documents the process most clearly. In its JavaScript SEO basics, it says Googlebot parses the response for URLs in the href attribute of HTML links and adds them to the crawl queue, then parses the rendered HTML for links again after rendering. Its link best practices put the rule in one line: generally, Google can only crawl your link if it is an <a> HTML element with an href attribute.

AI operators do not publish how their crawlers discover URLs. OpenAI, Anthropic and Perplexity document their agents and how they treat robots.txt, but not link discovery, at the time of writing. Two things are known, though:

• Most AI crawlers read raw HTML only. Vercel and MERJ's December 2024 analysis found that none of the major AI crawlers rendered JavaScript at the time, including OpenAI's crawlers, ClaudeBot and PerplexityBot. A link that exists only after scripts run is invisible to them. See do AI crawlers run JavaScript? for the detail.
• They hit a lot of dead URLs. In the same study, on nextjs.org, 34.82% of ChatGPT's crawler fetches and 34.16% of Claude's landed on 404 pages, compared with 8.22% for Googlebot. ChatGPT's fetches also hit redirects 14.36% of the time. That is one site, measured in late 2024, but it shows that every dead link a crawler finds costs a request that returns nothing.

So the safe design target is the least capable crawler: links that are present in the server's HTML, written as plain anchors, pointing at URLs that answer 200.

Which links crawlers can follow, and which they cannot

The difference is in the markup, not in how the link looks on screen. These all look the same to a visitor:

<!-- Crawlable: a real anchor with a real URL -->
<a href="/pricing">Pricing</a>
<a href="https://example.com/pricing">Pricing</a>
<a href="/pricing" onclick="track('nav')">Pricing</a>

<!-- Not reliably crawlable -->
<a onclick="goto('/pricing')">Pricing</a>
<span href="/pricing">Pricing</span>
<button onclick="location.href='/pricing'">Pricing</button>
<a href="javascript:goTo('pricing')">Pricing</a>
<a href="#/pricing">Pricing</a>

Google lists <a> without href, <span href>, onclick navigation and javascript: URLs as not recommended in its link guidance: Google may still try to parse them, but you should not rely on it. Adding an onclick to a normal anchor is fine, because the href is still there.

Hash routes (#/pricing) are their own problem. Google's JavaScript guide says that, to make sure Googlebot can parse and extract your URLs, you should not use fragments to load different page content, and it recommends the History API with real paths instead. Framework link components such as Next.js <Link> already render a real <a href>, so the usual cause is hand-built menus, cards with a click handler on a <div>, or a router configured in hash mode.

Buttons, "Load more" and infinite scroll

Google's pagination guidance says Google's crawlers do not click buttons and generally do not trigger JavaScript functions that need a user action. A product list that only grows when someone presses "Load more" shows a crawler the first batch and nothing else.

The fix is to give each page of results its own URL and link them with anchors:

<nav aria-label="Pagination">
  <a href="/shoes?page=1">1</a>
  <a href="/shoes?page=2">2</a>
  <a href="/shoes?page=3">Next page</a>
</nav>

You can keep the button for visitors and still render these links. Google's guidance also says not to use # fragments for page numbers, and that Google no longer uses rel="next" and rel="prev" (other search engines may).

Link text matters too

Google says to write descriptive anchor text and avoid "click here" or "read more", and that an image link uses the image's alt text as its anchor text. Link text is part of how machines understand the target page. Headings, landmarks and link text covers that side of structure.

Orphan pages and link depth

An orphan page has no internal link pointing to it. It may be in your sitemap, or linked from an old email campaign, but nothing on the site leads there. Google's own wording is that every page you care about should have a link from at least one other page on your site.

Common causes:

• A page was removed from the menu during a redesign but kept live.
• Landing pages built for ads and never linked from the site.
• Blog posts reachable only through a paginated archive that uses a "Load more" button.
• Category or tag pages that were deleted, leaving products with no path to them.

Link depth is the number of clicks from the homepage to a page. Google does not publish a depth limit and there is no documented rule like "three clicks or fewer". The practical point is simpler: a page buried behind long chains of pagination is found later and less often than one linked from the menu, a hub page or a related-posts block. Put important pages within a few clicks, and link from pages that are themselves well linked.

A sitemap helps a crawler find orphans, but it does not replace links. Google's pagination guidance suggests a sitemap (or a Merchant Center feed) to help Google find all the products on a site, alongside <a href> links between pages. XML sitemaps for AI discovery explains what a sitemap can and cannot do.

Broken links and redirect hops in navigation

A broken internal link points at a URL that returns 404, 410 or a server error. A crawler that follows it spends a request and gets nothing to read. Worse cases:

• Soft 404s. The URL answers 200 but shows "page not found". Google's status code documentation says Search Console reports these as soft 404 errors. A crawler that does not detect soft 404s reads the error page as content.
• Redirects to the homepage. Sending every dead URL to / gives the crawler a page unrelated to the one it asked for.
• Redirects inside navigation. A menu that links to http://example.com/about when the real page is https://www.example.com/about/ can send every visit through up to three hops: protocol, host and trailing slash.

Googlebot generally follows up to 10 redirect hops, according to the same page. OpenAI, Anthropic and Perplexity do not publish a hop limit at the time of writing. Redirects are fine for old URLs that people and other sites still use, but your own links should never need them. Point every internal link at the final URL, with the right protocol, host and trailing slash. Status codes, redirects and AI crawlers covers how each status is handled.

On that last row: Common Crawl's FAQ says CCBot honors nofollow on links, so an internal nofollow can stop it reaching a page. Internal links rarely need nofollow at all.

How to find broken internal links

You can find most problems with standard command-line tools. Run them against a staging copy if your production firewall rate-limits you.

Check what links a page offers in raw HTML

This lists every href in the HTML the server sends, which is what a non-rendering crawler sees:

curl -s https://example.com/ \
  | grep -oE '<a [^>]*href="[^"]*"' \
  | sed -E 's/.*href="([^"]*)".*/\1/' \
  | sort -u

If your main menu links are missing from this list but show up in the browser, the menu is built by JavaScript. Compare with the rendered page in your browser's developer tools; how to see your page as an AI crawler, with curl walks through the comparison.

Crawl the site and list broken links

wget can follow links like a simple crawler and report dead ones without saving pages:

wget --spider -r -l 4 -nd -nv -e robots=on \
  -o crawl.log https://example.com/
sed -n '/broken link/,$p' crawl.log

At the end of the log, wget prints "Found N broken links." followed by the dead URLs, which the sed line shows. It does not say which page links to each one, so search your templates or content for the URL to find the source. -l 4 limits depth to four clicks from the start page. A page that exists but never appears in this crawl is either an orphan or deeper than the limit. Because wget does not run JavaScript, it also shows you the site as a raw-HTML crawler sees it.

Check every sitemap URL for status and redirects

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+' | sed 's/<loc>//' \
  | while read -r url; do
      curl -s -o /dev/null -L \
        -w '%{http_code} %{num_redirects} %{url_effective}\n' "$url"
    done \
  | awk '$1 != 200 || $2 > 0'

Each line that prints is a sitemap URL that does not answer 200 directly: either an error or at least one redirect. A sitemap should list only final, live URLs.

Find orphan pages

Put your sitemap URLs in one file and the URLs your wget crawl found in another, both sorted, then compare:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+' | sed 's/<loc>//' \
  | sort -u > sitemap-urls.txt
grep -oE 'URL: ?[^ ]+' crawl.log | sed -E 's/^URL: ?//' \
  | sort -u > crawled-urls.txt
comm -23 sitemap-urls.txt crawled-urls.txt

The output is URLs in the sitemap that the crawl did not reach through internal links (within its depth limit). Both lists must use the same host and trailing-slash style, or every URL will look orphaned. Each one needs a link from a relevant page, or, if it should not exist, removal from the sitemap. In Google Search Console, the Page indexing report also lists URLs Google found that return 404 under "Not found (404)", which can trace back to an old internal link.

How to fix link problems for good

1. Use real anchors everywhere. Every navigational element that goes to another page should be an <a href> with a real path, rendered in the server HTML.
2. Link to final URLs. Use the canonical host, protocol and trailing-slash style. Root-relative links avoid host mistakes. Canonical tags and duplicate URLs covers picking one version.
3. Give paginated lists real pages. Keep "Load more" for visitors, but render numbered links to ?page=2 and beyond.
4. Fix links at the source. When you move a page, add the redirect for outside links, then update every internal link to the new URL.
5. Return honest status codes. A missing page gets a 404 or 410, not a 200 error page or a redirect to the homepage.
6. Check after every release. Run the wget crawl and sitemap check in CI or on a schedule, so a template change that breaks the footer gets caught in a day, not a quarter.

Frequently asked questions

Do AI crawlers follow internal links?

OpenAI, Anthropic and Perplexity do not document how their crawlers discover URLs. Following <a href> links in raw HTML and reading sitemaps are the standard methods, so plain HTML links are the safe assumption for every crawler.

Are JavaScript links bad for SEO and AI search?

Links inserted by JavaScript are fine for Google if they are real <a href> elements, because Googlebot renders pages. Most AI crawlers did not render JavaScript in Vercel and MERJ's 2024 measurement, so links that exist only after scripts run may be invisible to them.

Is a redirect in a menu link a problem?

It works, but it costs an extra request on every crawl and every visit, and chains add more. Change the link to point straight at the final URL.

Does a sitemap fix orphan pages?

It helps crawlers find them, but it does not give them internal links. Google says every page you care about should have a link from at least one other page on your site; a sitemap is an extra discovery aid, not a substitute.

How often should I check for broken internal links?

After every release that touches templates, navigation or URLs, and on a regular schedule otherwise. Automating the check catches breakage before crawlers record it.

Check your own pages

To see where your own site stands, start with the free check: it tests a single page and needs no account. For the whole site, a full audit runs deterministic checks across up to 500 pages, with human review and a PDF of the evidence and prioritized fixes.