Skip to content

React streaming SSR and isbot: which crawlers get full HTML

How isbot decides between onAllReady and onShellReady in React Router and Hydrogen, which AI crawlers it matches, and how to test both responses with curl.

By Crawlito · 9 min read

In short

  • React's docs suggest waiting for onAllReady, or awaiting stream.allReady, for crawlers, so they get the final HTML instead of a shell with Suspense fallbacks.
  • The default entry.server in React Router 7 and 8 calls isbot on the User-Agent header and waits for onAllReady only when it returns true; Shopify's Hydrogen template does the same with renderToReadableStream.
  • In isbot 5.2.3, the published GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User strings all match, but through generic patterns such as the word bot or a URL, not by crawler name.
  • A browser-style user agent that only adds a token such as Claude-User/1.0 does not match isbot 5.2.3, and gets the streamed shell.
  • In the default React Router entry, an error in any Suspense boundary sets the status to 500, so a bot that waited for onAllReady can receive a 500 for a page a browser received with a 200.
On this page

React can stream server-rendered HTML in two ways: send the shell as soon as it is ready (onShellReady) and stream Suspense content later, or wait until everything has rendered (onAllReady) and send the finished document. React Router, which succeeded Remix v2, and Shopify's Hydrogen template pick between the two with the isbot package: if isbot says the user agent is a bot, the crawler gets complete HTML; if not, it gets the streamed shell. The AI crawler user agents published by OpenAI and Perplexity match the current isbot, but only through generic patterns, so check your installed version and the exact strings rather than assuming.

This post is for developers on React Router 7 or 8, Remix v2, Hydrogen or a custom React server. For what streamed Suspense output looks like in Next.js and how its htmlLimitedBots list works, see Next.js SEO for AI crawlers. If your app has no server rendering at all, start with fixing the empty React SPA shell.

What React says about crawlers and onAllReady

The renderToPipeableStream reference has a section on waiting for all content to load for crawlers and static generation. onShellReady fires once the shell has rendered, while content inside <Suspense> may still be loading. onAllReady fires when everything is complete, and React says that if you start piping there, the stream is not progressive: it contains the final HTML.

The docs leave the bot check to you: their example has let isCrawler = // ... depends on your bot detection strategy .... They also name the cost. A crawler that waits has to wait for all data, some of which might be slow or fail, and React says you could choose to send the shell to crawlers too.

The web streams version, renderToReadableStream, works the same way with a promise: await stream.allReady before returning the response.

Here is what the two modes send for the same page. We rendered a heading, one Suspense boundary that resolves after 300 ms and one that rejects, with React 19.3.1 (inline scripts shortened):

html
<!-- onShellReady: what a browser gets -->
<h1>Shell title</h1>
<!--$?--><template id="B:0"></template><p>Loading...</p><!--/$-->
<!--$?--><template id="B:1"></template><p>Loading reviews...</p><!--/$-->
<script>…</script>
<div hidden id="S:0"><p>Real product description</p></div>
<script>…</script>

<!-- onAllReady: what a matched bot gets -->
<h1>Shell title</h1>
<!--$--><p>Real product description</p><!--/$-->
<!--$!--><template data-msg="…"></template><p>Loading reviews...</p><!--/$-->

In the shell version the description sits in a hidden div near the end, swapped into place by an inline script. In the complete version it is in position. The failed boundary shows its fallback in both and is left for the client to render; only the complete version marks it with <!--$!--> in the HTML, while the shell version keeps <!--$?--> and switches it with a script. A crawler that does not run JavaScript never sees those reviews either way.

How React Router and Hydrogen use isbot

At the time of writing, React Router's latest release is 8.5.1 and the 7.x line is at 7.18.5. The default Node entry has the same bot logic in both. React Router's entry.server.tsx docs say the file is optional on Node, where a default is used, that other runtimes such as Cloudflare need their own file, and that you can see the default with:

bash
npx react-router reveal

The default Node entry chooses the callback like this:

tsx
import { isbot } from "isbot";

export const streamTimeout = 5_000;

// inside handleRequest
let userAgent = request.headers.get("user-agent");

let readyOption: keyof RenderToPipeableStreamOptions =
  (userAgent && isbot(userAgent)) || routerContext.isSpaMode
    ? "onAllReady"
    : "onShellReady";

The web streams default in the same folder (entry.server.web.tsx) does the same with await body.allReady. Shopify's Hydrogen template, which pins React Router 7, has its own app/entry.server.tsx with the same logic:

tsx
if (isbot(request.headers.get("user-agent"))) {
  await body.allReady;
}

Remix v2's default entry also used isbot, with a helper that supports both the default export of isbot versions before 3.8.0 and the named isbot export of later ones. Whatever your framework, the decision rests on one function and one header.

SetupBot checkBot getsEveryone else gets
React Router 7 or 8, default Node entryisbot(userAgent)onAllReadyonShellReady
React Router 7 or 8, default web entryisbot(userAgent)await body.allReadyStreamed shell
Hydrogen templateisbot(...)await body.allReadyStreamed shell
Custom renderToPipeableStream serverWhatever you writeYour choiceYour choice

At the time of writing, @react-router/dev 8.5.1 depends on isbot ^5.1.40 (7.18.5 on ^5.1.11), its build adds isbot ^5 to your package.json if server rendering is on and the package is missing, and the Hydrogen template asks for ^5.1.36. A fresh install gets the latest 5.x, which was 5.2.3 when we tested; an older lockfile can pin an older one. Run npm ls isbot to see what you actually ship. Framework details for React Router are in React Router v7 SEO for AI crawlers, and Hydrogen's robots.txt and hosting in Hydrogen and Oxygen robots.txt.

Which AI crawlers isbot matches

The isbot README describes a regular expression built from a list of patterns, aimed at "good bots" that identify themselves in the user agent. It exports helpers to inspect a match: isbotMatches returns the substrings that matched and list holds the pattern parts. Version 5.2.0 added camel-case names (isBot, findBotMatches, createIsBotFromList) and kept the old ones working; the README says the old names will be dropped in a future version 6. The snippets below use the old names because they work in every 5.x release and match what React Router imports.

We ran isbot 5.2.3 against the strings the operators publish. OpenAI's bots page and Perplexity's crawler page both give example strings and say the version numbers may change. Anthropic names ClaudeBot, Claude-SearchBot and Claude-User but does not publish full strings, so we used minimal test strings for those.

CrawlerString testedisbot 5.2.3Matched because of
GPTBotOpenAI's published stringtruethe word bot and a URL
OAI-SearchBotOpenAI's published stringtruethe word bot and a URL
ChatGPT-UserOpenAI's published stringtruebot, a URL and User/1.0;
PerplexityBotPerplexity's published stringtruethe word bot, a URL and /1.0;
Perplexity-UserPerplexity's published stringtruea URL and User/1.0;
ClaudeBotMozilla/5.0 (compatible; ClaudeBot/1.0)truethe word bot and the short compatible shape
Claude-UserMozilla/5.0 (compatible; Claude-User/1.0)truethe short compatible shape
Desktop Chromea normal Chrome stringfalsenothing

None of these crawlers is in the list by name. The only AI-related names we found among its 206 patterns were claude-code/, openai/ and manus-user/. That works today, but it is fragile. We tested a browser-style string with a token added and no URL:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; Claude-User/1.0

isbot 5.2.3 returns false for it, so that request would get the streamed shell. We do not know Anthropic's real Claude-User string; the point is that a user-triggered fetcher with a browser-like string and no bot or URL in it would slip through.

How to test what your installed isbot matches

Run this from your project root so it uses the version in your node_modules:

bash
node -e '
const { isbot, isbotMatches } = require("isbot");
const v = require("isbot/package.json").version;
const uas = {
  GPTBot: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
  "ChatGPT-User": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",
  PerplexityBot: "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
  "Perplexity-User": "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",
  ClaudeBot: "Mozilla/5.0 (compatible; ClaudeBot/1.0)",
};
console.log("isbot", v);
for (const [name, ua] of Object.entries(uas)) console.log(name, isbot(ua), isbotMatches(ua));
'

Add the OAI-SearchBot string from OpenAI's page and any user agents you see in your own logs. Real strings from your logs are the best test, because they are what the function will see in production.

How to extend the bot check

If you want named AI crawlers to get complete HTML regardless of how the generic patterns evolve, build your own function with createIsbotFromList and use it in entry.server.tsx. Patterns are matched case-insensitively:

tsx
// app/lib/is-crawler.ts
import { createIsbotFromList, list } from "isbot";

const AI_CRAWLERS = [
  "gptbot", "oai-searchbot", "chatgpt-user",
  "claudebot", "claude-searchbot", "claude-user",
  "perplexitybot", "perplexity-user",
];

export const isCrawler = createIsbotFromList(list.concat(AI_CRAWLERS));

Then reveal the entry and swap the call (in a web streams entry or the Hydrogen template, swap the isbot(...) call before await body.allReady the same way):

tsx
import { isCrawler } from "./lib/is-crawler";

let readyOption: keyof RenderToPipeableStreamOptions =
  (userAgent && isCrawler(userAgent)) || routerContext.isSpaMode
    ? "onAllReady"
    : "onShellReady";

With this list, the browser-style Claude-User string above returns true. Two cautions. First, this only changes timing and order: the bot gets the same content a browser ends up with, not different content, so it stays clear of serving crawlers different content. Second, the isbot README itself says not to trust the user agent alone for allowlisting; here it only picks a rendering mode, which is harmless if a person spoofs it.

The cost of waiting: timeouts and status codes

Waiting has two side effects worth knowing before you add more crawlers to the check.

Timeouts. The default entry exports streamTimeout = 5_000 and aborts the render at streamTimeout + 1000. The docs say streamTimeout is how long the server waits for streamed promises before rejecting them. A matched bot can therefore wait up to about six seconds for its first byte when a loader is slow, and React's docs say that on abort it flushes the remaining fallbacks as HTML and leaves the rest to the client. In our React 19.3.1 test, an abort also called onError, so with the default entry a bot that hits the timeout gets a 500 as well. The web streams default uses the same timeout through AbortSignal.timeout; the Hydrogen template passes the request's own signal and sets no timeout of its own. Lower the timeout, or move slow, non-essential data out of the critical path.

Status codes. The default onError sets responseStatusCode = 500, and the response is built inside the ready callback. The web streams default and the Hydrogen template also set 500 in onError and build the response after await body.allReady. A browser's response is created in onShellReady, usually before a deferred boundary fails, so it gets 200. A bot's response is created in onAllReady, after the error, so it gets 500. We reproduced this with React 19.3.1: the same render produced a 200 in shell mode and a 500 in complete mode. If a reviews widget can fail, a crawler can see a server error on a page that looks fine in your browser. Run the curl comparison below on pages with optional data, and if a 500 appears only for the bot, adjust onError so that only errors before the shell is ready change the status (for example, set a flag in an onShellReady callback, which you can pass alongside onAllReady), or make the failing data source return an empty result instead of throwing.

How to compare both responses with curl

Fetch the page twice, once as a browser and once as a crawler, and look for the streaming markers React uses. Use a real GET: the default entry returns an empty body for HEAD, so curl -I tells you nothing about the HTML.

bash
URL='https://example.com/products/blue-mug'
BROWSER='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
BOT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'

for ua in "$BROWSER" "$BOT"; do
  echo "== ${ua:0:60}"
  curl -s -o page.html -w 'status %{http_code}, first byte %{time_starttransfer}s\n' -A "$ua" "$URL"
  echo "pending boundaries: $(grep -o '<template id="B:[0-9]*"' page.html | wc -l)"
  echo "hidden streamed blocks: $(grep -o '<div hidden id="S:[0-9]*"' page.html | wc -l)"
  echo "client-rendered fallbacks: $(grep -o '<!--\$!-->' page.html | wc -l)"
done

What to expect:

  • Browser: pending boundaries and hidden blocks greater than zero if the page uses Suspense, and a fast first byte.
  • Matched bot: zero pending boundaries and zero hidden blocks, a slower first byte, and your main text in position.
  • A bot with the same counts as the browser: isbot did not match that string, or a custom entry ignores it. Fix the check as above.
  • Client-rendered fallbacks above zero in the bot response, or a 500 for the bot only: a boundary failed or timed out on the server. Find it before a crawler does.

A CDN in front of the app can cache one variant and serve it to everyone, so test the origin as well. Seeing your page as an AI crawler with curl has more checks, including word counts.

Frequently asked questions

Does React itself detect bots?

No. React's docs show onAllReady and stream.allReady for crawlers but leave detection to your code. React Router and Hydrogen use the isbot package for it.

Is GPTBot treated as a bot by React Router's default entry?

With isbot 5.2.3, yes: OpenAI's published GPTBot, OAI-SearchBot and ChatGPT-User strings all match, so they get onAllReady. They match through generic patterns, not by name, so test the version in your lockfile.

Is the streamed shell bad for crawlers that do not run JavaScript?

The text is still in the response, but resolved Suspense content sits in hidden elements near the end and is moved into place by inline scripts. No AI operator documents how its crawler treats that, so the complete document is the safer response for crawlers.

Can I just use onAllReady for everyone?

You can, but every visitor then waits for the slowest data before seeing anything, which is the cost React warns about. A narrower fix is to keep critical text outside Suspense, so the shell already contains it.

Does waiting for onAllReady fix data fetched in useEffect?

No. Effects run only in the browser, so that content is missing from both responses. Load it in a route loader or a server component instead.

Check what a crawler receives

After changing the bot check, run the curl comparison above, then the free single-page check, which needs no account, on a page that uses Suspense. For a whole React Router or Hydrogen site, the audit covers up to 500 pages and comes as a PDF with evidence and prioritized fixes.

Filed under Rendering and content. Topics: react, ssr, rendering, ai-crawlers, javascript