Do AI crawlers run JavaScript? What is known, and how to test

Googlebot runs JavaScript. Most AI crawlers, in the public measurements available, do not. Google documents that it renders pages in an evergreen Chromium, while the most widely cited measurement of AI crawlers, Vercel and MERJ's December 2024 study, found that none of the major ones it tracked executed JavaScript. Until an operator documents otherwise, assume an AI crawler reads only the HTML your server sends, and put the content that matters there.

This post covers what Google says, what the Vercel study actually measured and where its limits are, a crawler-by-crawler table, and a test you can run on your own pages today.

What Google says about Googlebot and JavaScript

Google is the one operator that documents its rendering in detail. Its JavaScript SEO basics guide describes three phases: crawling, rendering and indexing. Pages that return a 200 status go into a render queue. When Google's resources allow, a headless, evergreen Chromium renders the page and executes the JavaScript, and Google indexes the rendered HTML.

Two details matter in practice:

• Rendering is queued. Google says a page may sit in the render queue for a few seconds, but it can take longer. Content in the initial HTML is available at crawl time; content that needs JavaScript waits.
• Google still recommends server rendering. The same guide says server-side rendering or pre-rendering is still a great idea, because it makes a site faster for users and crawlers, and because not all bots can run JavaScript.

This also covers Google's AI features. Google says that to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Search with a snippet, and that there are no additional technical requirements. So for Google, a JavaScript-rendered page that Googlebot renders and indexes correctly is in the same position as any other indexed page. The open question is everyone else.

What the Vercel study measured

The most cited evidence is Vercel's post The rise of the AI crawler, dated 17 December 2024 and written by Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl and Ryan Siddle of MERJ. It is worth knowing exactly what it looked at before you quote it.

The data. Crawler traffic mainly on nextjs.org and across the Vercel network, with the findings checked against two job-board sites, Resume Library and CV Library. For the month it reported, Googlebot made about 4.5 billion fetches across Search and Gemini, GPTBot 569 million, Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million.

The main finding. None of the major AI crawlers rendered JavaScript at the time. The post names OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot, and says Common Crawl's CCBot does not render pages either.

The telling detail. ChatGPT's and Claude's crawlers did download JavaScript files, 11.50% and 23.84% of their requests respectively, but did not execute them. Fetching a script file is not the same as running it, so seeing .js requests from GPTBot in your logs does not mean it saw your client-rendered content.

The renderers. The study said Gemini uses Googlebot's infrastructure and so gets full rendering, and that AppleBot renders JavaScript through a browser-based crawler.

What the study does not tell you

• It is a snapshot from late 2024. Crawlers change. Cite it as "as of December 2024", not as a permanent fact.
• It does not cover every agent. ChatGPT-User is the only user-triggered agent it names. Claude-User, Perplexity-User and browser-based agents that drive a real browser may behave differently. Anthropic now documents three bots (ClaudeBot, Claude-SearchBot and Claude-User), and the 2024 data does not cover the two newer ones.
• It is one network. The sample is large, but it is Vercel's traffic, not the whole web.

More recent statements point the same way. A Vercel knowledge-base guide dated 1 October 2026 says background crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fetch HTML without executing it, and Wix's documentation says most AI crawlers don't run JavaScript and read only the HTML the server sends. Neither is a statement from the crawler operators themselves.

Which crawlers render JavaScript: a summary table

"Not documented" means the operator's own crawler page says nothing either way at the time of writing. That is different from "no".

Apple also warns that blocking JavaScript or CSS files in robots.txt can stop Applebot rendering a page properly, which is good advice for every renderer: do not disallow /static/ or /_next/ if crawlers need those files.

Why the safe assumption is "server HTML only"

If a crawler does not run JavaScript, it sees the response body your server returns and nothing else. On a client-side rendered app that can be an empty shell:

<body>
  <div id="root"></div>
  <script src="/assets/index-3f2a.js"></script>
</body>

A person sees a full page a second later. A crawler that does not execute scripts sees no words at all. We cover this failure in depth in client-side rendering and the empty HTML problem.

Fully client-rendered apps are the obvious case, but partial gaps are more common and easier to miss:

• Data fetched in the browser. Prices, reviews, product specs or article bodies loaded with fetch after the page mounts, for example in a React useEffect.
• Head tags set by script. A title, meta description or canonical changed client-side after a generic one was served.
• JSON-LD injected by JavaScript. Google says Search can process structured data that is in the DOM when it renders the page, including markup added with Google Tag Manager. A crawler that does not render never sees it.
• Content behind interaction. Tabs, accordions or "load more" sections that request their text only when clicked. See lazy loading, tabs and accordions.

A Next.js streaming caveat

Server rendering frameworks mostly solve this, with one subtlety. In the Next.js App Router, content inside a Suspense boundary that resolves at request time is streamed in the same response, but it arrives near the end inside a hidden element (for example <div hidden id="S:0">) and is moved into place by an inline script. The text is in the HTML, just out of position and hidden. For user agents on its htmlLimitedBots list, Next.js waits for the full render and sends one complete document instead, and at the time of writing the default list does not include GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. How each AI crawler treats hidden, out-of-order text is not documented. The careful options are to keep key content outside request-time Suspense boundaries, or to extend htmlLimitedBots (setting it replaces the default list, so add to it rather than overwrite it). More in Next.js and AI search.

How to test whether your content needs JavaScript

You need a terminal and about five minutes per page template. Test one page of each type: home, product, article, category.

1. Fetch the raw HTML as an AI crawler would

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
curl -sL -A "$UA" -o raw.html -w '%{http_code}\n' https://example.com/pricing

The status code should be 200. Now pick a sentence that only appears in the main content, such as a price or the second paragraph of an article, and search for it:

grep -c "Billed annually at" raw.html

0 means that text is not in the server's HTML. A crawler that does not run JavaScript will not see it. Two cautions: pick a phrase without apostrophes or ampersands, which are often written as HTML entities, and remember that a match can come from a JSON blob inside a <script> tag rather than visible markup. If the count is above zero, open raw.html and check where the match sits.

Spoofing the user agent tests the HTML, not the crawler's real network path. A firewall can treat the real GPTBot differently from your laptop. If raw.html is a Cloudflare challenge page, the response carries a cf-mitigated: challenge header. Check the headers of the same GET request:

curl -s -o /dev/null -D - -A "$UA" https://example.com/pricing | grep -i cf-mitigated
``` That is a separate problem, covered in [why ChatGPT can't see your website](/blog/why-chatgpt-cant-see-my-website).

### 2. Compare raw and rendered word counts

Render the same page with headless Chrome, then count the words in both files. On macOS the Chrome binary sits inside the app bundle (`/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`); on Linux it is usually `google-chrome`. `--dump-dom` prints the DOM once the page has loaded, so content fetched later, or only on scroll or click, can still be missing from `rendered.html`.

```bash
google-chrome --headless --dump-dom https://example.com/pricing > rendered.html

Save this as words.py. It ignores script, style, noscript and template content and prints a word count:

import sys
from html.parser import HTMLParser

class Text(HTMLParser):
    skip = {"script", "style", "noscript", "template"}
    def __init__(self):
        super().__init__()
        self.depth = 0
        self.words = []
    def handle_starttag(self, tag, attrs):
        if tag in self.skip:
            self.depth += 1
    def handle_endtag(self, tag):
        if tag in self.skip and self.depth:
            self.depth -= 1
    def handle_data(self, data):
        if not self.depth:
            self.words += data.split()

p = Text()
p.feed(open(sys.argv[1], encoding="utf-8", errors="ignore").read())
print(len(p.words))

python3 words.py raw.html
python3 words.py rendered.html

If the raw count is close to the rendered count, your content does not depend on JavaScript. If raw is a small fraction of rendered, the difference is what non-rendering crawlers miss. A walkthrough of reading the two versions side by side is in raw HTML vs rendered HTML.

3. The quick browser check

In Chrome DevTools, open the command menu, run "Disable JavaScript" and reload. Whatever disappears is content that depends on scripts. Use "View page source" rather than the Elements panel when you want the raw HTML: the Elements panel shows the live DOM after scripts have run.

How to fix content that only exists after JavaScript

1. Render on the server or at build time. Server-side rendering or static generation puts the full text in the first response. Most frameworks support this: Next.js Server Components, Nuxt, SvelteKit, Astro, Remix. The trade-offs are in SSR, SSG and CSR compared.
2. Move key data fetches to the server. If a product page renders on the server but loads its price and description in the browser, fetch them on the server instead.
3. Output head tags and JSON-LD in the server HTML. Title, meta description, canonical, robots meta and structured data should all be in the initial response.
4. Put tab and accordion text in the HTML. Hide it with CSS until clicked if you like, but do not fetch it on click.
5. Use prerendering if you cannot change the app. A build-time prerender step can generate static HTML for each route and serve it to everyone. Serving rendered HTML only to bots is dynamic rendering, which Google calls a workaround and not a recommended solution; it recommends server-side rendering, static rendering or hydration instead. If you do use it, the bot version must match what users see, or it can be treated as cloaking.

After the change, run the curl test again and confirm your sentence appears in raw.html.

Frequently asked questions

Does ChatGPT render JavaScript when it crawls?

OpenAI's crawler documentation does not say. Vercel and MERJ's December 2024 analysis found that GPTBot, OAI-SearchBot and ChatGPT-User did not execute JavaScript, so text that only appears after scripts run was not visible to them at the time.

Does Googlebot render JavaScript for AI Overviews?

AI Overviews and AI Mode link to pages that are indexed in Google Search, and Googlebot renders JavaScript with an evergreen Chromium before indexing. Rendering is queued, though, and Google still recommends server-side rendering or pre-rendering.

If GPTBot downloads my JavaScript files, does it run them?

Not according to the 2024 Vercel study. ChatGPT's and Claude's crawlers fetched JavaScript files but did not execute them, so a .js request in your logs is not evidence that client-rendered content was read.

Is JSON-LD added by Google Tag Manager visible to AI crawlers?

Google can read structured data that is present in the rendered DOM. A crawler that does not run JavaScript will not see JSON-LD injected by a tag manager, so output it in the server HTML.

Will server-side rendering get my site cited in AI answers?

No one can promise that. Server rendering makes your content readable to crawlers that do not run JavaScript, which removes one barrier; whether an assistant cites a page depends on many things outside your site.

Check your own pages

The test above takes a few minutes per page template. Run it on your home page and one page of each other template, and fix the ones where key text is missing from the raw HTML. If you would rather start without a terminal, the free single-page check needs no account; for a whole site of up to 500 pages, an audit adds human review and a PDF of prioritized fixes.