Next.js SEO for AI crawlers: rendering, metadata and robots

Next.js is a good starting point for AI search. In the App Router, pages are Server Components by default and are prerendered to HTML on the server, so the text is in the response before any script runs. The problems come from a few specific patterns: data fetched in the browser, content streamed late through Suspense, metadata that streams after the head, and robots or noindex settings generated from code nobody rechecks.

This guide goes through each one, with a way to check it and a fix. It applies to the App Router in Next.js 15 and 16. This site runs on Next.js too, so where it helps we show what we did.

What the App Router sends by default

The Next.js docs on Server and Client Components say layouts and pages are Server Components by default. On the first load, Next.js uses the Server Components and the Client Components together to prerender HTML on the server. The browser shows that HTML as a non-interactive preview, then JavaScript hydrates it.

That matters because most AI crawlers read only the first response. The Vercel and MERJ analysis from December 2024 found that none of the major AI crawlers they measured rendered JavaScript, naming OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Googlebot is different: Google's JavaScript SEO guide says it renders pages with an evergreen Chromium, after a queue that can take a few seconds or longer. For background, see do AI crawlers run JavaScript?.

So a Next.js page is readable to an AI crawler when the words are in the HTML that the server sends. Here is how each common pattern fares:

Why 'use client' pages that fetch in useEffect look empty

'use client' is not the problem on its own. A Client Component is still rendered to HTML on the server for the first load. The problem is where the data comes from. An effect runs only in the browser, after hydration, so the server renders whatever the component shows before the effect runs.

'use client'
import { useEffect, useState } from 'react'

export default function PricingPage() {
  const [plans, setPlans] = useState<Plan[] | null>(null)
  useEffect(() => {
    fetch('/api/plans').then((r) => r.json()).then(setPlans)
  }, [])
  if (!plans) return <p>Loading plans...</p> // this is all the server HTML contains
  return <PlanTable plans={plans} />
}

A crawler that does not run JavaScript reads "Loading plans..." and nothing else. This is the Next.js version of the empty HTML problem.

The fix is to fetch on the server and pass the data down. Keep the page a Server Component and move only the interactive part into a Client Component:

// app/pricing/page.tsx (Server Component)
import { PlanTable } from './plan-table' // 'use client' inside, for the toggles

export default async function PricingPage() {
  const plans = await getPlans() // runs on the server
  return <PlanTable plans={plans} />
}

The table's text is now in the prerendered HTML, and its buttons still work after hydration. Data a visitor never needs to see before interacting, such as a cart or account menu, can stay client-side.

Suspense and streaming: in the response, but not in place

Streaming is where people talk past each other. The Next.js streaming guide explains that the App Router sends one HTTP response in chunks aligned with Suspense boundaries. The static shell (layouts, navigation and Suspense fallbacks) goes first. When a boundary resolves, React streams its finished HTML later in the same response.

So the text is not missing. But it does not arrive where the reader expects. The guide describes it: the resolved content arrives inside <div hidden id="S:0">, with an inline script that swaps it into a placeholder marked <template id="B:0">. A crawler that reads the raw HTML without running that script sees your skeleton in the article's position and the real article in a hidden element near the end of the document.

The loading.js reference says streaming "does not impact SEO", and that statement is about Google and similar search crawlers. How each AI crawler's HTML-to-text extraction treats hidden, out-of-order content is not documented by any of the operators. We would not rely on it for your main content.

What to do:

1. Keep the text that matters, such as the article body, product name, price and description, outside Suspense boundaries that resolve at request time, or make it part of the static shell by caching its data.
2. Use loading.js and <Suspense> for parts nobody needs to cite: recommendations, personalised widgets, live stock counts.
3. Or tell Next.js to send AI crawlers a fully rendered document, as described in the next section.

One more streaming detail: once streaming has started, the status code is already 200. The streaming guide says a notFound() that fires mid-stream cannot become a 404, so Next.js injects <meta name="robots" content="noindex"> instead, and a mid-stream redirect() becomes a client-side redirect. If a missing product should return a real 404 or a moved page a real 301, check for it before the first Suspense boundary, or handle it in next.config redirects or Proxy, which the guide says run before the page renders. With Cache Components enabled, the prerendered shell is sent first, so a request-time check inside the page still returns 200; use Proxy for those.

The metadata API and streamed metadata

Next.js generates <title>, the meta description, canonical links and robots meta tags from the metadata export or generateMetadata. Two things to know.

First, metadata can be streamed. The generateMetadata reference says that when generateMetadata depends on request-time data, its result can be sent after the initial UI, and the tags are appended to the <body>. Next.js says it has verified that bots which run JavaScript and inspect the full DOM, such as Googlebot, read this correctly. For user agents it classifies as HTML-limited bots, it waits for generateMetadata before starting the response and puts the tags in the <head>.

Second, the default HTML-limited list does not include the AI crawlers. In Next.js 16.4 it matches Google crawlers whose names end in -Google or start with Google- (such as AdsBot-Google and Google-InspectionTool), Chrome-Lighthouse, Bingbot, applebot, Twitterbot, Slackbot, facebookexternalhit and others, but not GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. Main Googlebot is not on it, because it runs JavaScript. The list lives in html-bots.ts and can change between releases.

If your metadata is static or prerendered, this does not affect you: the docs say it is then included in the initial HTML. If it is dynamic, you can add AI crawlers with the htmlLimitedBots option. The docs say setting it overrides the default list, so keep the default and append to it. In the streaming guide's own test with a Twitterbot user agent, the server holds the response and sends the whole document in one burst, which also puts Suspense content in position. Other parts of the docs say the rest of the page can still stream once metadata resolves, so confirm the result with the curl check below.

// next.config.ts
import type { NextConfig } from 'next'

// Default list as shipped in Next.js 16.4 (html-bots.ts). Re-copy it when you upgrade.
const NEXT_DEFAULT =
  '[\\w-]+-Google|Google-[\\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight'

const AI_CRAWLERS =
  'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot'

const nextConfig: NextConfig = {
  htmlLimitedBots: new RegExp(`${NEXT_DEFAULT}|${AI_CRAWLERS}`, 'i'),
}

export default nextConfig

The trade-off: those bots wait for the full render before the first byte, so slow data sources make their responses slower.

A robots meta tag from the metadata API is just as easy to leak as one from any other platform. robots: { index: false } set from an environment variable on staging and copied to production hides the live site. See the leftover noindex.

app/robots.ts and app/sitemap.ts

The robots.txt file convention lets you return a MetadataRoute.Robots object from app/robots.ts, and Next.js serves it as /robots.txt. It is cached by default unless it uses a request-time API or a dynamic config option. Here is a version that allows search crawlers and opts out of training:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'CCBot', 'Google-Extended', 'Applebot-Extended'],
        disallow: '/',
      },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

In Next.js 16.4 it renders as plain robots.txt, with one group per rule. An array of user agents becomes several User-Agent lines sharing one set of rules, which RFC 9309 treats the same as separate groups:

User-Agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: CCBot
User-Agent: Google-Extended
User-Agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

One trap: a crawler that finds a group naming it ignores the * group. If you add { userAgent: 'OAI-SearchBot', allow: '/' }, OAI-SearchBot no longer sees your /admin/ disallow. Repeat shared rules in every named group. How robots.txt groups work covers this in detail. Also make sure app/robots.ts is the only source of the file (no leftover public/robots.txt or app/robots.txt), and fetch the live /robots.txt after deploying: a CDN or host can serve its own.

app/sitemap.ts works the same way, returning a MetadataRoute.Sitemap array. Build it from the same source as your pages so new pages appear automatically, and set lastModified from real content dates rather than the build time. A single sitemap file can hold at most 50,000 URLs, per sitemaps.org; larger sites can split it with generateSitemaps.

JSON-LD in a Next.js page

The Next.js JSON-LD guide recommends rendering structured data as a native <script type="application/ld+json"> in a layout or page, not with next/script, which is meant for executable JavaScript. It also warns that JSON.stringify does not sanitise strings, and suggests replacing < with its Unicode escape, <. This is the component this site uses:

export function JsonLd({ data }: { data: object }) {
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(data).replace(/</g, '\\u003c') }}
    />
  )
}

Render it from a Server Component so the markup is in the prerendered HTML, and build it from the same data as the visible page so the name, price or date always match. Avoid placing it inside a Suspense boundary that streams late. Common mistakes are covered in JSON-LD errors.

How to check a Next.js page as an AI crawler

Request the page with a crawler's user agent and look at the raw response. Use the production URL, because next dev behaves differently from a production build.

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
URL='https://example.com/pricing'

echo "1. Main text present?"
curl -s -A "$UA" "$URL" | grep -c 'a phrase from the main content'

echo "2. Streamed, hidden Suspense blocks:"
curl -s -A "$UA" "$URL" | grep -o '<div hidden id="S:[0-9]*"' | head

echo "3. Title and canonical inside <head>:"
curl -s -A "$UA" "$URL" | perl -0ne 'print "$1\n" if /<head(?:\s[^>]*)?>(.*?)<\/head>/s' \
  | grep -oE '<title>[^<]*</title>|<link rel="canonical"[^>]*>'

If step 1 returns 0, the content is fetched in the browser. If step 2 finds hidden blocks containing your main text, move that text out of Suspense or add the crawler to htmlLimitedBots. If step 3 prints nothing, your metadata is streaming into the body for that user agent. Run the same commands with Twitterbot/1.0 as the user agent to see what a fully blocking render looks like. Our curl guide has more variations.

Also check what runs before your pages. A Next.js Proxy (formerly Middleware) rule, a host firewall or a CDN bot setting can block or challenge AI crawlers before your app runs, and none of that shows up in your code.

Frequently asked questions

Is Next.js good for AI search?

Its defaults are a good start. App Router pages are Server Components that are prerendered to HTML on the server, so their text is in the first response. Problems come from fetching data in the browser, streaming key content through Suspense, and generated robots or noindex settings.

Does 'use client' hide content from AI crawlers?

Not by itself. Client Components are still prerendered to HTML on the server for the first load. Content goes missing when it depends on data fetched in useEffect or on a user interaction.

Do AI crawlers see content inside Suspense boundaries?

The content is in the same HTTP response, but inside a hidden div near the end, with only the fallback in its visual position until an inline script runs. No AI operator documents how its crawler handles that, so keep important text outside request-time Suspense boundaries.

Should I add GPTBot and ClaudeBot to htmlLimitedBots?

If your pages use dynamic generateMetadata or stream important content, it is a reasonable change: those bots then get metadata in the head, and in Next.js's own bot test the whole document arrives at once. Check the result with curl, and keep the default list in the regex, because setting the option replaces it.

Does the Pages Router behave differently?

The Pages Router does not use Server Components or streamed generateMetadata: getServerSideProps and getStaticProps resolve data before the page is rendered, so the Suspense and streamed metadata issues above mostly do not apply. Fetching in useEffect has the same problem in both routers.

Next step

To see what an AI crawler receives from one of your Next.js pages, run the free check, which needs no account: it reports the robots.txt rule for each AI crawler, robots meta and headers, and whether the main text is in the raw HTML. For a whole site, the audit compares raw and rendered HTML across up to 500 pages.