Skip to content

Astro SEO for AI crawlers: islands, client:only and sitemaps

Astro serves static HTML by default, which suits AI crawlers. Where client:only, server islands, SSR routes and missing site config still hide content.

By Crawlito · 6 min read

In short

  • Astro prerenders every page to HTML by default, and framework components without a client directive render as HTML with no JavaScript, so most Astro content is visible to crawlers that do not run scripts.
  • A component with client:only skips server rendering entirely, so crawlers that read raw HTML see only its fallback slot, or nothing.
  • Server islands (server:defer) ship a placeholder and a script in the initial HTML and fetch the real content afterwards, so their content is missing from the raw HTML.
  • The @astrojs/sitemap integration requires site in astro.config and cannot list dynamic routes rendered on demand.
  • On-demand pages must set Astro.response.status or return a Response for missing content, or a missing product can answer 200.
On this page

Astro is one of the friendliest frameworks for AI crawlers, because it renders pages to plain HTML by default and only adds JavaScript where you ask for it. Content in .astro components, and framework components without a client:* directive, is in the HTML a crawler downloads. The problems come from four specific features: client:only components, server islands, routes rendered on demand, and integrations that need configuration such as the site URL. Check those and an Astro site is usually in good shape.

This matters because many AI crawlers read the HTML your server sends and do not run JavaScript, as covered in do AI crawlers run JavaScript. Anything Astro only renders in the browser is invisible to them.

How Astro renders pages: static by default, on demand when asked

Astro's on-demand rendering guide says the whole site is prerendered by default. That is the 'static' output mode: every page becomes an HTML file at build time.

You can change that in two ways:

  • Per page: export const prerender = false makes one page or endpoint render on demand, on each request.
  • Whole site: output: 'server' in astro.config.mjs makes on-demand rendering the default, and export const prerender = true opts single pages back into prerendering.

Either way, rendering on demand needs an adapter for your host. Both modes produce HTML on the server, so both are readable by crawlers. The difference that matters for crawlers is not static versus server, but what happens in the browser afterwards.

Islands and client directives: what ends up in the HTML

Astro's directives reference defines how each framework component (React, Vue, Svelte and others) is delivered:

UsageServer HTMLJavaScriptCrawler without JS sees
<Widget /> (no directive)RenderedNoneFull content
<Widget client:load />RenderedHydrates on page loadFull content
<Widget client:idle />RenderedHydrates after initial loadFull content
<Widget client:visible />RenderedHydrates when scrolled into viewFull content
<Widget client:only="react" />SkippedRenders in the browser onlyFallback slot, or nothing
<Widget server:defer />PlaceholderFetched after loadFallback slot, or nothing

The client:load, client:idle and client:visible directives are safe for content: the component is rendered on the server first and only made interactive later. The two rows in bold are where content goes missing.

The client:only pitfall

The docs say client:only "skips HTML server rendering, and renders only on the client". You also have to name the framework, as in client:only="react", because Astro never runs the component on the server and cannot infer it.

People reach for client:only when a component touches window or localStorage during render and breaks the build. The quick fix works, and the component's text disappears from the HTML. If that component holds a pricing table, product specs or an FAQ, AI crawlers that read raw HTML never see it.

Better fixes, in order:

  1. Make the component safe to render on the server. Move browser-only code into an effect or event handler, then use client:load or client:visible.
  2. Split it. Render the content as plain HTML in an .astro component, and keep only the interactive control as a client:only island.
  3. If it must stay client-only, give it a real fallback. A child element with slot="fallback" is rendered in the HTML until the component loads. Put the essential text there, not a spinner.
astro
---
import PriceCalculator from "../components/PriceCalculator.jsx";
---
<PriceCalculator client:only="react">
  <div slot="fallback">
    <p>Starter: $29 a month. Team: $99 a month. Billed annually.</p>
  </div>
</PriceCalculator>

Server islands load their content after the page

server:defer turns a component into a server island. Astro's server islands guide explains the delivery: at build time Astro omits the component and injects a script in its place, and in the browser that script requests the island's HTML from a separate endpoint. The fallback slot is rendered with the page initially and replaced when the content arrives.

That is ideal for personalised bits such as an avatar or a cart count. It is the wrong choice for anything you want crawlers to read, because the raw HTML holds only the fallback.

How to check what crawlers see on an Astro page

Look at the HTML the server sends, not the page in your browser:

bash
curl -s https://www.example.com/pricing | grep -c "Team: \$99"

A count of 0 for text you can see in the browser means it is rendered in the browser. Then search your source for the two directives that skip server rendering:

bash
grep -rn "client:only\|server:defer" src/

Review each match and ask whether the content inside should be readable by crawlers. Raw vs rendered HTML covers the comparison in more depth.

Three other places hide content in Astro projects more often than people expect:

  • Data fetched in the browser. A React or Vue island that calls an API in an effect renders an empty shell on the server, even with client:load. Fetch the data in the Astro frontmatter instead and pass it to the component as props, so it is in the HTML.
  • Tabs and accordions built as islands. If only the active tab is rendered on the server, the other panels exist only after a click. Render all panels in the HTML and toggle visibility with CSS or the hidden attribute.
  • Embedded third-party widgets. Reviews, store locators and comparison tables loaded from another service by script are not part of your HTML at all.

If you are deciding how much of a site to prerender, SSR, SSG and CSR for AI search compares the trade-offs.

robots.txt and sitemaps in Astro

Sitemap integration

The @astrojs/sitemap integration is added with npx astro add sitemap. Two limits matter:

  • It needs site. You must set site in astro.config.mjs to a full URL starting with http:// or https://, because sitemaps need absolute URLs.
  • It cannot list on-demand dynamic routes. Dynamic routes built with getStaticPaths() are included. Routes like src/pages/products/[slug].astro rendered on demand are not. If your product or article pages are rendered on demand, generate their sitemap yourself from the same data source.

The build writes sitemap-index.xml, which points to sitemap-0.xml and further numbered files on large sites. Use the filter option to leave out pages such as thank-you or search pages.

js
// astro.config.mjs
import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";

export default defineConfig({
  site: "https://www.example.com",
  integrations: [
    sitemap({ filter: (page) => !page.includes("/thank-you/") }),
  ],
});

robots.txt

The simplest option is a static public/robots.txt, which Astro copies as is. The sitemap docs also show an endpoint at src/pages/robots.txt.ts that builds the Sitemap: line from your site value, so the URL is never out of date:

ts
import type { APIRoute } from "astro";

const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /

Sitemap: ${sitemapURL.href}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemapURL = new URL("sitemap-index.xml", site);
  return new Response(getRobotsTxt(sitemapURL));
};

Add groups for specific AI crawlers here if you need them. robots.txt for AI crawlers covers which tokens to use.

Status codes, canonicals and other Astro details

  • 404 pages. A src/pages/404.astro file builds to 404.html, which most hosts pick up for unknown URLs. Confirm with curl that your host returns it with a 404 status, not 200. On on-demand pages, a missing record does not trigger it automatically. Set Astro.response.status = 404 in the page frontmatter, or return new Response(null, { status: 404 }), otherwise a deleted product answers 200 with an empty template. Astro's docs note that response headers can only be set at the page level, not inside components.
  • Redirects. In on-demand pages, use return Astro.redirect("/new-url", 301) for moved content, so crawlers get a permanent redirect rather than a page that redirects with JavaScript.
  • Canonical tags. Astro does not add them for you. Build them in your layout from Astro.site and Astro.url.pathname, and keep the trailingSlash setting consistent with the URLs in your sitemap.
  • View transitions and client routing. These change how users move between pages but not the HTML each URL returns. Check one URL with curl to confirm.

Frequently asked questions

Is Astro good for SEO and AI crawlers?

Its defaults are. Astro prerenders pages to HTML and ships no JavaScript for components without a client directive, so content is in the raw HTML. Problems come from client:only, server islands and missing configuration.

Does client:visible hide content from crawlers?

No. client:visible renders the component on the server and only delays hydration until it scrolls into view. The HTML is already there for crawlers.

Why are my product pages missing from the Astro sitemap?

If they are dynamic routes rendered on demand, the sitemap integration cannot generate entries for them. Prerender them with getStaticPaths() or build a sitemap endpoint from your product data.

Do I need an adapter for robots.txt in Astro?

No. A file in public/robots.txt, or a src/pages/robots.txt.ts endpoint that is prerendered, works on a fully static site.

Next step

Run curl against your most important templates and look for text that only appears in the browser. The free check does a similar test for one page with no account, and a full audit checks rendering, robots.txt and sitemaps across up to 500 pages.

Filed under Platform guides. Topics: javascript, rendering, ssr, sitemaps, robots-txt, raw-html