Skip to content

Discovered, currently not indexed? Check crawl access

Investigate URLs Google has discovered but not crawled by checking public access, useful internal links, URL expansion, and server capacity with real logs.

By Crawlito · 6 min read

In short

  • Discovered but not indexed indicates that Google knows the URL without a recorded crawl in that report.
  • A new sitemap entry does not establish that the page was fetched.
  • Server response evidence and the paths that expose URLs help prioritize a crawl investigation.
On this page

If a page is “Discovered - currently not indexed,” confirm that its public URL works and examine how Google could reach it through the site. Then review server capacity, request failures, and large families of unnecessary URLs. Google's status definition says the URL is known but has not yet been crawled, often because the expected load led Google to defer the request. Search Console’s Page indexing definitions

Define the set before changing the whole site

Select a small group of important affected URLs and write down how they were published. Are they new guides from one release, products added through an import, or filters generated from every possible attribute combination? A publishing event can create a few useful pages or thousands of nearly equivalent addresses. The intervention should reflect what the site actually produced.

Include one older working page from the same section and one new page that has already been crawled, if available. Record publication times, template, sitemap membership, and the path from a public hub. This helps reveal whether the excluded group shares a route defect or simply appeared in a much larger publication burst.

Check the specific URL in Search Console and preserve the report date. The absence of a last-crawl date fits this status, but a more recent URL inspection may show a changed state. Avoid treating a historical export as a live queue. Keep the original sample fixed long enough to compare later observations without silently replacing its difficult cases with successful ones.

Verify that a first visit can retrieve the page

Open each important example without a signed-in session. Capture the response status, final URL, and main content. A route can work through in-app navigation while failing when opened directly. A product imported into the CMS may also have a preview page without a published public route. Establish public availability before asking why a crawler has not visited.

bash
curl -sSL --max-time 30 -D guide-headers.txt   -o guide.html https://example.com/guides/new-device

Inspect redirects and the body, including any challenge or regional selection screen. Check robots.txt for the exact host and path, plus the page's own indexing directives when it can be fetched. A blocked route, a slow response, and a readable guide are different observations with different owners. The HTTP status guide helps describe what the server actually delivered.

Repeat the request at an ordinary interval if the page appears unstable. Record timeouts as timeouts rather than giving them a fabricated status. A local network fault should be compared with application and edge evidence before being attributed to the origin. The purpose is to find a reproducible obstacle, not to infer the entire crawler experience from one client.

Start at the homepage or a relevant public hub and follow ordinary anchors to the new page. Look at the link in the delivered HTML. A clickable card whose destination exists only inside JavaScript state may work for a person while supplying a weaker discovery path for clients reading the initial document.

Check the newest and oldest entries on the section index. A “show more” control can make the latest page visible while leaving all earlier pages reachable only after interaction. An archive whose pagination accidentally repeats the first page can similarly strand later entries. The orphan page guide provides a way to verify whether important pages have useful incoming links.

Add links where they serve a reader. A device setup guide belongs beside the device's specifications and troubleshooting material. Repeating a large list of unrelated links in every footer increases noise without explaining which page answers the visitor's question. Use clear labels and verify that the final destination is the intended canonical URL.

Look for crawl demand created by the site itself

Review which URL families appear in logs, sitemaps, and link crawls. A newly published filter interface might expose combinations of color, size, order, view, and session identifiers. The count can grow much faster than the number of products a visitor can buy. Identify the generator and decide which combinations represent useful stable pages.

Compare the important discovered set with those expanded families. A large volume of parameter requests can justify examining navigation and caching, but it does not prove that any one guide was deferred for that reason. Keep the diagnosis tied to observed request patterns, failure rates, and the site's intended URL inventory. The faceted navigation guide explains how to investigate these paths without hiding useful category pages indiscriminately.

A calendar that creates links indefinitely into the future is another useful test case. Choose a future boundary where the publication has no events and inspect whether the interface still emits new crawlable month links. Set a meaningful navigation horizon based on the actual content and user need. Verify a real future event remains reachable after the change.

Review capacity and verify a measured repair

Use request logs and your existing monitoring to examine latency, timeouts, rate limits, and server errors around normal crawler activity. Verify crawler identity before grouping requests by bot name. A cache miss that launches an expensive catalog query can affect direct page requests differently from your own warm browser test.

Google documents the role of serving capacity and crawl demand in its crawl budget guidance for large sites. Compare those concepts with the scale and symptoms you actually observe. A small site with a broken route needs that route repaired; a large site repeatedly timing out during crawler activity may need a more substantial delivery change. Google’s crawl budget guidance

Choose an intervention with a visible acceptance test. A repaired hub should link to the intended guide in its HTML. A fixed route should return the article on a direct request. A cache change should reduce the observed expensive origin work under a controlled test. Confirm those outcomes before examining whether Google subsequently crawls the sample.

Track the original URLs through later inspections and verified request logs. A crawl can happen before a report updates, and indexing can remain a separate decision after a successful fetch. Preserve these milestones individually. This gives the publisher a useful record of what became accessible and what the search engine later chose to do with it.

Compare publishing bursts with ordinary releases

Choose one quiet publishing day and one large import from your own release history. Compare the number of new public URLs, origin response times, and verified crawler errors during each period. Keep traffic and caching conditions in the comparison where the monitoring supports them.

If the import exposed many unused combinations or caused an expensive cache rebuild, you now have a concrete operational change to investigate. If both periods served the sampled pages reliably, retain that evidence and focus on the remaining discovery and content questions. A large URL count alone cannot establish which request Google deferred or why.

Frequently asked questions

Does adding a URL to a sitemap force a crawl?

A sitemap supplies discovery information. Google decides when to crawl and which pages to index.

Should I increase the server's bot rate limit immediately?

Review actual rate-limit events and resource use first. If a limit is the observed obstacle, adjust it with the hosting owner and verify the site's capacity.

They can provide useful navigation and context. Test that readers can reach the page naturally and that the link points to its maintained public address.

Check one important discovered URL

Run the free page check on a representative page. Combine the response with its internal link path and server evidence before choosing a crawl-related fix.

Filed under Access and robots.txt. Topics: search-console, crawl-budget, internal-links