Skip to content

Next.js ?_rsc= URLs in Search Console: why Googlebot crawls them

Why Googlebot requests Next.js ?_rsc= URLs, what the _rsc parameter does, and whether to leave them, block them in robots.txt or reduce Link prefetching.

By Crawlito · 9 min read

In short

  • The _rsc query parameter is a hash Next.js adds to App Router requests for React Server Component payloads, so CDNs that ignore Vary still cache RSC and HTML responses under different keys.
  • Next.js Link prefetches routes automatically in production as links enter the viewport, and Vercel's guidance says link prefetch payloads can be spotted by the ?_rsc= query parameter.
  • The first load of a Next.js page returns server-rendered HTML, and Next.js documents that later navigations fetch only the RSC payload, so a crawler that fetches the clean URL does not need the separate ?_rsc= requests to see the content.
  • Next.js says the _rsc parameter must stay in the CDN cache key, so never strip or redirect it away to tidy Search Console; use robots.txt or fewer prefetches instead.
  • Google does not document how it treats ?_rsc= URLs, so verify in your own logs that Googlebot still gets a 200 HTML response for the clean URL before and after any change.
On this page

URLs ending in ?_rsc= followed by a short random-looking value come from the Next.js App Router, not from your links. When a <Link> prefetches a route, or the router navigates on the client, Next.js asks the server for the React Server Component (RSC) payload of that route and adds an _rsc hash to the query string so caches keep that response separate from the HTML. Google renders pages with JavaScript, so it can see those requests too. In most cases you can leave them alone. If they clutter your reports or logs, disallow the pattern in robots.txt or reduce prefetching, but never strip the parameter at your CDN.

This post covers what Next.js and Google document about these URLs, what they do not, and how to check in your logs that page rendering is unaffected. General parameter policy is covered in URL parameters and AI search, and the risks of blocking files a renderer needs are in blocking CSS and JavaScript in robots.txt.

What the _rsc parameter is

Next.js documents the parameter in its CDN caching guide. App Router responses vary by several request headers, and Next.js sends a Vary header naming them:

HeaderWhat it signals
rscReturn an RSC payload instead of HTML
next-router-state-treeThe client's current router state
next-router-prefetchThis is a prefetch request
next-router-segment-prefetchThe specific segment being prefetched
next-urlThe intercepted URL, only for interception routes

Many CDNs do not honour Vary without extra configuration. Next.js handles this with _rsc, which its documentation describes as a hash of the relevant request header values that acts as a cache key, so that different response variants get different cache keys. The same page says the parameter "must be included in the cache key" and that the rsc header must reach the server, because otherwise the server returns HTML where the client router expects RSC data.

So /pricing?_rsc=1x2y3 is not a different page. It is the same route, requested as a different kind of response.

The response itself is not HTML. The Next.js glossary defines the RSC payload as a compact representation of the rendered Server Components tree: the rendered result of Server Components, placeholders for Client Components and the props passed between them. The browser's router uses it to update the page without a full reload.

Why Googlebot ends up requesting ?_rsc= URLs

Two documented facts explain where these URLs come from.

  1. Link prefetches automatically. The Next.js prefetching guide says prefetching runs automatically in production: as each <Link> enters the viewport, Next.js prefetches the route behind it. The Link reference says the default ("auto" or null) prefetches static routes in full and dynamic routes down to the nearest loading.js boundary.
  2. Prefetches carry the parameter. Vercel's guide to RSC payload size says that for link prefetching you can find RSC payloads "by looking for the ?_rsc=... query parameter".

Google's JavaScript SEO basics say a headless Chromium renders the page and executes its JavaScript. A page full of <Link> components therefore runs the same router code in Google's renderer as in a browser, and any prefetch requests that code makes come from Google. Google does not document whether its renderer triggers every viewport-based prefetch, so your logs are the evidence.

What is not documented: Google has not published how it treats ?_rsc= URLs it has seen, whether it queues them as pages, or how it decides to show them in the Page indexing report. Next.js does not document crawler behaviour either. Treat anything beyond the two facts above as an observation from your own data, not a rule.

What a plain fetch of a ?_rsc= URL returns

The response depends on the request headers, which is why these URLs confuse people when they test them by hand.

RequestWhat to expect, based on the Next.js docs
Clean URL, no rsc headerServer-rendered HTML for the page
?_rsc= URL with the rsc header and a correct hashThe RSC payload, not HTML
rsc header with a missing or wrong _rsc valueBy default a 307 redirect to the URL with the correct hash
?_rsc= URL without the rsc headerHTML, because the rsc header is what asks for the payload

The first row comes from the prefetching guide: on the first navigation the browser fetches the HTML, JavaScript and RSC payload, and later navigations fetch only the RSC payload. How that payload is embedded in the HTML is explained in the RSC payload and self.__next_f.push. The redirect in the third row is the default described in the CDN guide, and it can be turned off with experimental.validateRSCRequestHeaders.

The last row matters for Search Console. If a crawler later requests a ?_rsc= URL as an ordinary page, without the router's headers, it should get the same HTML as the clean URL. That is a near duplicate with an extra parameter, and a canonical tag on the page pointing to the clean URL signals which address you prefer. In Next.js you set it with alternates.canonical in your metadata.

You can see the difference yourself:

bash
# The page as a crawler first sees it
curl -sI https://example.com/pricing | grep -iE '^(HTTP|content-type|vary)'

# The same URL as the client router asks for it
curl -sI -H 'rsc: 1' 'https://example.com/pricing?_rsc=test' | grep -iE '^(HTTP|content-type|location)'

Expect text/html on the first request, usually with a vary header that names rsc. The second may return the payload or a 307 to a corrected _rsc value, depending on your version and settings. Hosting layers can change these results, so trust what your own server returns.

Should you block _rsc URLs in robots.txt?

You have three reasonable options and one bad one.

OptionWhat it doesWhen it fits
Leave themNothing changes. Google decides how to treat the URLsSmall and medium sites where the URLs are a reporting nuisance, not a load problem
Disallow the patternGooglebot stops requesting matching URLsLarge sites where these requests are a visible share of Googlebot's crawling
Reduce prefetchingFewer ?_rsc= requests from every visitor, including crawlersPages with long lists of links, such as archives and category pages
Strip or redirect _rsc at the CDNBreaks the cache key Next.js relies onNever

The last row is the trap. A redirect rule that removes _rsc "to clean up parameters" can lead a CDN to serve a cached HTML response to the client router, or the reverse. The CDN guide is explicit that _rsc is mandatory for prefetch flows and must stay in the cache key.

If you choose to disallow, Google's robots.txt specification supports * as a wildcard, matches case-sensitively and applies the most specific matching rule. Because _rsc can be the first parameter or a later one, cover both positions:

User-agent: Googlebot
Disallow: /*?_rsc=
Disallow: /*&_rsc=

Put the lines in the group your Googlebot rules already use, or in User-agent: * if you want every crawler to skip them. Do not add a new User-agent: Googlebot group just for these two lines if your other rules live under *: the same specification says only one group applies to a crawler, and Googlebot follows its most specific group and ignores *, so it would stop obeying your other rules. The wildcard guide explains why /*_rsc= alone would also match paths you did not intend. In the App Router, a robots.ts file can generate the same rules:

tsx
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', disallow: ['/*?_rsc=', '/*&_rsc='] },
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Two consequences to accept before you ship it. First, a disallowed URL can still be listed in Search Console as blocked by robots.txt, which is expected and harmless for URLs you never wanted indexed; see indexed, though blocked by robots.txt for the rarer case where one is indexed anyway. Second, Google's crawl budget guide says blocking URLs does not move crawl budget to other pages unless Google is already at your site's crawl capacity limit, so do not expect a blocked pattern to speed up indexing elsewhere.

Blocking _rsc is different from blocking /_next/static/. The static folder holds the JavaScript and CSS the page needs, and blocking it can stop Google rendering the page properly. Prefetch _rsc requests are for routes the visitor has not opened yet, and the current page's server-rendered content is already in its HTML.

One version caveat: the CDN guide notes that when Cache Components is enabled, segment-level prefetches already use pathname-based URLs such as /page.segments/_tree.segment.rsc, and the Next.js team is working on moving fully to pathname-based keys and removing _rsc. At the time of writing that is still in design, but check your logs for .rsc paths too, because the patterns above do not match them.

How to reduce prefetching without losing navigation speed

If the requests come from pages with hundreds of links, cut them at the source. The prefetching guide documents three tools:

  • prefetch={false} on a <Link> stops prefetching on viewport entry and on hover. Static routes are then fetched on click, and dynamic routes wait for the server before navigating.
  • Hover-triggered prefetch. The guide's example sets prefetch={active ? null : false} and flips active on onMouseEnter, so only links a person points at are prefetched. A renderer that never hovers makes no prefetch.
  • router.prefetch() from useRouter lets you prefetch only the routes you choose.
tsx
<Link prefetch={false} href={`/blog/${post.slug}`}>
  {post.title}
</Link>

None of these change the <a href> in your HTML, so crawlers still find the link. The guide suggests this for large lists of links such as an infinite-scroll table, and footer links are its example of links that may not need prefetching.

How to confirm in logs that rendering is unaffected

The question that matters is whether Googlebot still gets your real pages. Check that directly rather than reading the Page indexing report counts.

  1. Confirm the requests are real Googlebot. Filter your access logs for _rsc= and Googlebot's user agent, then verify a sample of IPs with a reverse and forward DNS lookup, as Google's Googlebot verification guide describes.
  2. Compare with the clean URLs. For the same paths, list Googlebot requests without _rsc. They should return 200 with text/html if you log content type.
bash
grep 'Googlebot' access.log | grep -v '_rsc=' | awk '{print $9}' | sort | uniq -c
grep 'Googlebot' access.log | grep '_rsc=' | awk '{print $9}' | sort | uniq -c

The field number for the status code depends on your log format. Clean URLs answering 200 mean Google is fetching your pages normally. A run of 307 on _rsc requests is the documented hash correction, not an error.

  1. Run URL Inspection on the clean URL. Google's URL Inspection tool live test shows a screenshot of the rendered page, the HTML and the page resources loaded. The main content should be present whether or not the prefetches were allowed.
  2. Repeat after any robots.txt change. Google generally caches robots.txt for up to 24 hours, so compare a few days of logs, not a few minutes.

The Next.js readiness guide covers what the App Router sends to crawlers that do not run JavaScript at all, which is a separate question from prefetch traffic.

Frequently asked questions

Are ?_rsc= URLs duplicate content?

They are the same route as the clean URL, requested as a different response type. If a crawler fetches one without the router's headers it gets the same HTML, and a canonical tag pointing to the clean URL states your preference.

Will blocking _rsc in robots.txt hurt my rankings?

Google does not document any effect, and no ranking outcome can be promised either way. The current page's server-rendered content is in its HTML, so blocking the pattern should not change how Google renders the clean URL; confirm with URL Inspection.

Can I remove the _rsc parameter with a redirect?

No. Next.js says the parameter must stay in the CDN cache key and is mandatory for prefetch requests, so removing it can serve HTML to the client router or RSC data to a browser.

Do AI crawlers request ?_rsc= URLs?

Only a crawler that runs your JavaScript would trigger prefetches, and at the time of writing OpenAI, Anthropic and Perplexity do not document JavaScript rendering for their crawlers. Check your own logs for _rsc= by user agent rather than assuming.

Check the page a crawler actually gets

The ?_rsc= requests are a side effect. What matters is the HTML your clean URLs return. Run the free check on an important page to see whether its main text is in the raw HTML and whether robots.txt lets the main AI crawlers fetch it.

Filed under Rendering and content. Topics: nextjs, url-parameters, search-console, robots-txt, googlebot