Raw HTML vs rendered HTML: seeing what a crawler sees

Raw HTML is the document your server returns for a URL, byte for byte. Rendered HTML is what that document becomes after a browser downloads it, runs its JavaScript and builds the page you see. Googlebot indexes the rendered version; the major AI crawlers measured so far read only the raw one. So the gap between the two is the part of your page that some crawlers never see.

This post shows how to look at each version, how to put a number on the difference, and how to tell a harmless gap from one that hides your content.

What raw HTML and rendered HTML actually are

When a browser requests a page, the server sends back an HTML response. That response is the raw HTML, sometimes called the initial HTML or the server response. It may already contain the full article, or it may be a near-empty shell like this:

<!doctype html>
<html lang="en">
  <head>
    <title>Loading…</title>
    <script src="/static/js/main.4f2a1c.js" defer></script>
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

The browser then parses the response, fetches scripts and stylesheets, runs the JavaScript, calls any APIs the scripts ask for, and inserts the results into the page. The document that exists after all that is the rendered HTML, more precisely the rendered DOM. In the example above, the rendered page might hold a full product description, a price and a set of reviews, none of which exist in the response the server sent.

The two are the same only when the server sends the finished page and scripts add nothing important. That is what server-side rendering and static generation produce. With client-side rendering, the gap can be the whole page. The SSR, SSG and CSR comparison covers the trade-offs between those approaches.

Which crawlers see which version

Two points about that table. First, Google renders, but rendering is queued: Google says a page may wait a few seconds in the render queue, or longer, and it still calls server-side rendering or pre-rendering "a great idea". Second, the Vercel data is a snapshot from late 2024 of a specific set of crawlers. It found that ChatGPT's and Claude's crawlers did download JavaScript files but did not execute them. It did not separately measure user-triggered fetchers such as ChatGPT-User, Claude-User or Perplexity-User, and OpenAI, Anthropic and Perplexity do not document JavaScript rendering on their crawler pages. The safe assumption is that the raw HTML is all an AI crawler gets. The full picture is in do AI crawlers run JavaScript?.

How to see the raw HTML

View-source in the browser

Open view-source:https://example.com/page, or press Ctrl+U (Cmd+Option+U on macOS). This shows the response the browser received for that URL, before any script ran. Use Ctrl+F to search it for a sentence from the middle of your main content. If the sentence is not there, a crawler that does not run JavaScript cannot read it.

Two caveats: view-source uses your browser's cookies and user agent, so a logged-in or personalised session can see a different response from a crawler; and it shows the response for the main document only, not data fetched later by scripts.

curl, with a crawler's user agent

curl fetches the raw response with no browser involved, and lets you send a crawler's user agent string. That matters because some firewalls and CDNs answer bots differently from browsers. The first command saves the plain response, the second repeats the request with GPTBot's user agent, and the third counts how many lines of the result contain a sentence you copied from the page:

curl -sL --compressed https://example.com/page -o raw.html

curl -sL --compressed \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/page -o raw-gptbot.html

grep -c "a sentence from your main content" raw-gptbot.html

A count of 0 means the sentence is not in the raw HTML, or is not there in that exact form. Servers often encode punctuation as entities (an apostrophe can arrive as '), so pick a run of plain words without quotes or symbols.

The user agent string is OpenAI's documented example from its crawler documentation; the version number may change. A spoofed user agent from your own IP address is an approximation: a firewall that verifies crawler IP ranges can treat the real GPTBot differently from your laptop. The step-by-step version, with headers and redirect checks, is in how to see your page as an AI crawler, with curl.

How to see the rendered HTML

The DevTools Elements panel

Open Chrome DevTools (F12) and look at the Elements panel. It shows the live DOM: the page after JavaScript has run, and after anything you have clicked, scrolled or expanded. This is the version closest to what a person sees, and roughly what Googlebot reads once rendering finishes. Do not mistake it for the raw HTML. Search here and in view-source for the same sentence, and the difference between the two results is the gap.

Disabling JavaScript

The quickest visual test: in DevTools, open the Command Menu (Ctrl+Shift+P, or Cmd+Shift+P on macOS), type "javascript", choose Disable JavaScript and reload. Chrome's documentation notes that JavaScript stays disabled in that tab for as long as DevTools is open. What you see now is roughly the raw HTML, drawn with CSS. A blank page, a spinner or a "please enable JavaScript" message means a non-rendering crawler gets the same nothing.

Headless Chrome from the command line

To save the rendered DOM to a file, run Chrome headless with --dump-dom. On Linux:

google-chrome --headless --dump-dom https://example.com/page > rendered.html

On macOS, call the binary inside the app bundle:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --headless --dump-dom https://example.com/page > rendered.html

This prints the DOM once the page has loaded. Content that loads only after scrolling or a click will not be in it, which is also true for most crawlers.

Search Console's URL Inspection

For Google specifically, URL Inspection shows what Googlebot got. On an indexed URL, View crawled page shows the HTTP response and the HTML Google received, with page resources under More info. Test live URL, then View tested page, fetches the page now and adds a screenshot of the rendered page, the HTTP headers, JavaScript console output and the resources loaded; Google says screenshots are available only in a live test (URL Inspection help). Google's JavaScript troubleshooting guide recommends using URL Inspection or the Rich Results Test to check "if the rendered HTML has all content you expect". The Rich Results Test does not need Search Console access, works on any page an anonymous visitor can reach, and its code view uses the rendered source.

Remember what this tells you: what Google saw after rendering. A page can look perfect here and still be empty to every AI crawler that reads only the raw HTML.

A quick metric: the raw to rendered text ratio

Searching for one sentence answers a yes or no question. To compare pages, or a whole site, count words instead:

ratio = words of visible text in the raw HTML ÷ words of visible text in the rendered page

In the browser console

Paste this into the DevTools console on the page you want to measure. It fetches the raw HTML again, parses it without running scripts, and compares its text with the rendered page.

const raw = await (await fetch(location.href, { cache: "no-store" })).text();
const doc = new DOMParser().parseFromString(raw, "text/html");
doc.querySelectorAll("script, style, noscript, template").forEach((n) => n.remove());
const textOf = (root) => {
  const walker = doc.createTreeWalker(root, NodeFilter.SHOW_TEXT);
  const parts = [];
  while (walker.nextNode()) parts.push(walker.currentNode.nodeValue);
  return parts.join(" ");
};
const words = (s) => (s.match(/\S+/g) || []).length;
const rawWords = words(doc.body ? textOf(doc.body) : "");
const renderedWords = words(document.body.innerText);
console.log({ rawWords, renderedWords, ratio: (rawWords / renderedWords).toFixed(2) });

The raw side joins text nodes with spaces, so minified markup such as <h1>Title</h1><p>Body</p> still counts as two words. The snippet uses your browser's cookies and user agent, so treat it as a check of your own rendering, not of what a firewall serves to bots.

From the command line

With raw-gptbot.html from curl and rendered.html from headless Chrome, this Python script counts text outside scripts and styles in both:

python3 - raw-gptbot.html rendered.html <<'EOF'
import sys
from html.parser import HTMLParser

SKIP = {"script", "style", "noscript", "template"}

class Text(HTMLParser):
    def __init__(self):
        super().__init__()
        self.skip = 0
        self.parts = []
    def handle_starttag(self, tag, attrs):
        if tag in SKIP:
            self.skip += 1
    def handle_endtag(self, tag):
        if tag in SKIP and self.skip:
            self.skip -= 1
    def handle_data(self, data):
        if not self.skip:
            self.parts.append(data)

def words(path):
    parser = Text()
    parser.feed(open(path, encoding="utf-8", errors="replace").read())
    return len(" ".join(parser.parts).split())

raw, rendered = words(sys.argv[1]), words(sys.argv[2])
print(f"raw={raw} rendered={rendered} ratio={raw / rendered:.2f}" if rendered else "rendered page has no text")
EOF

What counts as a meaningful gap

There is no published standard for this ratio, so read these bands as a rule of thumb for where to look, not as pass or fail:

A ratio can mislead in both directions. A cookie banner, chat widget or "recommended for you" carousel added by script lowers it without hiding anything important. A long footer and mega menu in the raw HTML can push it up while the article itself is missing. In the console version, collapsed accordions can even push the ratio above 1, because innerText skips hidden text. Some frameworks also stream late content into hidden elements near the end of the raw response: Next.js, for instance, sends resolved Suspense content inside a <div hidden> with a script that moves it into place, so the words are in the raw HTML but out of position. How crawlers treat that hidden text is not documented.

So after the number, check the parts that carry meaning. Each of these should be in the raw HTML:

1. The <title> and the H1, with the same wording you see on screen.
2. The first paragraphs of the main content, and any facts people search for: prices, specifications, opening hours, addresses.
3. Internal links in the main navigation and body, as real <a href> elements, not click handlers.
4. The meta description, canonical tag and robots meta tag, with the same values as in the rendered DOM.
5. JSON-LD structured data, if you use it. Markup injected by a tag manager exists only after rendering.

If something on that list appears only in the rendered version, a crawler that reads raw HTML sees the page without it, or sees different metadata from the one you tested. Content behind tabs and "load more" buttons needs its own check, which lazy loading, tabs and accordions walks through.

How to close the gap

The fix is to put the important content in the server response:

• Render on the server or at build time. Server-side rendering and static generation send the finished HTML. Google's dynamic rendering page recommends server-side rendering, static rendering or hydration instead of workarounds.
• Keep data fetching on the server. In React frameworks, content fetched in a client-side effect after load is missing from the raw HTML even when the rest of the page is server-rendered.
• Move metadata and JSON-LD into the server template instead of setting them with scripts.
• Treat dynamic rendering as a stopgap when a rebuild is not possible: serving crawlers a prerendered copy works only if the content matches what people see. Google calls dynamic rendering a workaround and not a recommended solution. Prerendering and dynamic rendering covers when each one helps.

Then measure again with the same commands, and compare the ratio before and after.

Frequently asked questions

Is view-source the same as what Googlebot sees?

No. View-source shows the raw HTML, which is what Googlebot fetches first. Googlebot then renders the page and indexes the rendered HTML, which is closer to the DevTools Elements panel. URL Inspection in Search Console shows Google's own view.

Do AI crawlers see the rendered HTML?

At the time of writing, assume not. Vercel's December 2024 analysis found that none of the major AI crawlers it measured executed JavaScript, and OpenAI, Anthropic and Perplexity do not document rendering support. Google renders pages for Search, and Apple says Applebot may render in a browser.

What is a good raw to rendered text ratio?

Close to 1 means the server already sends the content. There is no official threshold, but a ratio well below 0.5 usually means the main content depends on JavaScript, and the next step is to check whether the heading, main text and metadata are in the raw HTML.

Why does my page look fine in URL Inspection but not in AI tools?

URL Inspection shows the page after Google has rendered it. Crawlers that read only the raw HTML never run the scripts that add your content, so the same page can be complete for Google and empty for them.

Check one page now

Start with the page that matters most: run the curl and view-source checks above, then put the same URL through the free check, which tests one page with no account needed. If many templates look affected, a full-site audit covers up to 500 pages and comes with evidence and prioritized fixes.