Service workers and SEO: crawlers only ever make a first visit
Service worker SEO in one rule: a crawler never has your worker installed, so it gets the network response. See what that means for app shells and how to test it.
By Crawlito · 9 min read
In short
- A service worker only controls a page after an earlier visit registered it; MDN notes that a document starts life with or without a service worker and keeps that for its lifetime.
- A crawler that sends one GET request and runs no scripts never has your worker installed, so it always receives whatever your server or CDN returns over the network.
- Google documents that its renderer does not retain state across page loads, clearing cookies, Local Storage and Session Storage, so plan as if every Googlebot render is a first visit.
- In an app shell PWA the network response can be an empty shell that the worker serves from cache and the client fills with content, so a crawler gets only what the server returns.
- Test with curl and a fresh browser profile, or with the DevTools Bypass for network option, because a browser you have used before shows you the worker's version of the page.
On this page
For SEO, a service worker changes nothing about what a crawler receives: the crawler only ever gets the network response. A service worker starts controlling pages only after an earlier visit has registered and activated it, and a client that sends a single GET request and runs no scripts never gets that far. If your network response is a full HTML page, the worker is harmless. If it is an empty app shell that the worker fills from its cache, crawlers see the empty shell.
This post covers the service worker side of the problem: the lifecycle rule behind it, the PWA patterns that break it, and how to test without your own browser's worker getting in the way. For the general empty-HTML problem in client-side apps, see client-side rendering and the empty HTML problem.
Why a crawler never meets your service worker
A service worker is a script the browser installs for your origin. It sits between the page and the network and can answer requests from its own cache. MDN's guide to using service workers describes the order of events:
- The page loads from the network and its JavaScript calls
navigator.serviceWorker.register('/sw.js'). - The worker receives an
installevent, usually used to fill a cache with the files the app needs offline. - The worker activates.
- From then on,
fetchevents fire for documents inside the worker's scope and for the resources they request.
The key detail is when control starts. MDN says that documents already open when the worker activates have to be reloaded to be controlled, "because a document starts life with or without a service worker and maintains that for its lifetime." The web.dev service worker lifecycle article puts it the same way: you need to refresh the page to see the worker's effects. A worker can call clients.claim() to take over pages that are already open, but by then the HTML of that page has already arrived from the network.
Now look at it from a crawler's side. A crawler that does not run JavaScript fetches the HTML and never executes the register() call, so no worker is ever installed for it. Even when a crawler runs JavaScript and the register() call succeeds, the page that made the call was already delivered by the network. The first load of any page in a fresh client is always answered by the network, which means your server, origin cache or CDN.
| Client | Runs your register() call? | Who answers the page request |
|---|---|---|
| Returning visitor in a browser | Yes, on an earlier visit | Your service worker |
| First-time visitor | Yes, but only after this page loaded | The network |
| AI crawler fetching raw HTML | No | The network |
| Googlebot rendering the page | It may run the script, but state is not kept between loads | The network, for the HTML Google indexes |
Which AI crawlers render JavaScript at all is covered in do AI crawlers run JavaScript?. For this problem it hardly matters: rendering or not, the HTML document comes from the network.
What Google documents about stateless rendering
Google's guide to fixing Search-related JavaScript problems does not mention service workers. What it does say is that its Web Rendering Service (WRS) "does not retain state across page loads": Local Storage and Session Storage are cleared, and so are HTTP cookies. It also says Googlebot caches aggressively and that WRS may ignore caching headers, which can lead it to use outdated JavaScript or CSS.
Whether a worker registered during one render survives into the next is not documented. The safe reading of the documented behaviour is to treat every Googlebot render as a first visit by a browser that has never seen your site. Google's JavaScript SEO basics names the app shell model directly, describing it as a setup where "the initial HTML does not contain the actual content", and still recommends server-side rendering or pre-rendering because not every bot can run JavaScript.
Four PWA patterns that hide content from crawlers
Each of these looks fine in the browser you develop in, because that browser has the worker installed.
1. App shell served from the network
The web.dev PWA architecture guide describes the app shell model as one where the shell is cached, then served, and content is loaded on the client side. The service worker for that model often answers every navigation with the cached shell:
// sw.js: answer every navigation with the precached app shell
self.addEventListener('fetch', (event) => {
if (event.request.mode === 'navigate') {
event.respondWith(
caches.match('/app-shell.html').then((shell) => shell || fetch(event.request))
);
}
});For a returning visitor this is fast. The problem is what the server returns when the worker is not there. If /products/blue-chair on the network also returns the shell (because the server rewrites every route to app-shell.html), a crawler gets a page with a header, a footer and a loading placeholder, and the product text exists only in an API response the client fetches later.
2. Templating inside the service worker
The same web.dev guide describes streaming and service worker includes, where the worker splits a page into pieces with different caching needs and stitches them back together. If you build it so that the server returns only a content fragment or JSON for each route and the worker wraps it in the cached header and footer, a crawler requesting the URL from the network gets the fragment: no <title>, no meta description, no canonical tag and often no navigation links.
3. Different responses for navigation preload
Navigation preload lets the browser start the network request while the worker boots. MDN's NavigationPreloadManager page says the browser sends a Service-Worker-Navigation-Preload header with those requests, with a default value of true, so servers can tell them apart and, for example, send only part of the page. That is fine as long as a request without the header still gets the full page. Crawlers never send it. MDN also notes that if the two responses differ, the server must send Vary: Service-Worker-Navigation-Preload so caches keep them apart; without it, a CDN can store the partial response and serve it to everyone, crawlers included.
HTTP/2 200
content-type: text/html; charset=utf-8
vary: Service-Worker-Navigation-Preload4. Offline fallbacks and stale precaches
An offline fallback page shows visitors a friendly screen when the network fails. Crawlers do not get it. When your origin times out or cannot be reached, the crawler sees the real failure, while your team, testing in browsers with the worker installed, may see a cached page or the fallback. Note that MDN's fetch() reference says the fetch() promise does not reject on HTTP error statuses such as 404 or 504, so a worker only hides a 503 if its code checks response.ok and falls back, or answers from the cache first.
A precache works the other way round. Visitors with an old worker can keep seeing old HTML from the cache until the worker updates, while crawlers fetch the current network version. If an assistant quotes text you think you removed, check the network response before blaming the worker. That is a different problem with different causes, covered in CDN caching and stale page content.
| Pattern | What visitors with the worker see | What a crawler gets |
|---|---|---|
| App shell with client-side content | Full page from cache plus API data | Empty shell |
| Service worker templating | Assembled page | A fragment without head tags |
| Preload-specific responses | Full page | Full page, unless a cache mixed them up |
| Offline fallback | Friendly offline screen or cached page | The real timeout or error status |
| Stale precache | Old HTML | Current network HTML |
How to test what a crawler receives
The rule for every test: take your own service worker out of the picture. A browser you have used on the site has the worker installed and will answer from it.
Fetch the network response with curl
curl has no service worker, so it shows exactly what the network returns. Search for a sentence you expect on the page and for the head tags:
curl -s https://example.com/products/blue-chair | grep -c "Solid oak frame"
curl -s https://example.com/products/blue-chair | grep -iE "<title>|rel=\"canonical\"|name=\"description\""If you use navigation preload, compare the response with and without the header. The response without the header is the one crawlers get:
curl -s -o plain.html https://example.com/products/blue-chair
curl -s -o preload.html -H "Service-Worker-Navigation-Preload: true" https://example.com/products/blue-chair
wc -c plain.html preload.html
curl -sI https://example.com/products/blue-chair | grep -i "^vary"How to see your page as an AI crawler with curl has more variations, including crawler user agents.
Load the page in a fresh browser profile
To see the rendered first visit, start Chrome with an empty profile so there is no registration, cache or storage from earlier visits. On macOS:
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
--user-data-dir="$(mktemp -d)" https://example.com/products/blue-chairIn your normal browser, Chrome DevTools offers the same effect from the Application panel. Chrome's PWA debugging documentation describes a Bypass for network checkbox in the Service workers pane that forces requests to the network, and an Unregister link that removes the worker. To wipe everything from earlier visits, the Application panel's Storage pane has a Clear site data button, described in Chrome's cache storage documentation; older Chrome pages call the same place the Clear storage tab. web.dev also notes that a shift-reload bypasses the service worker entirely.
Compare with Google's view
For Googlebot, use URL Inspection in Search Console and read the rendered HTML, as described in the URL Inspection rendered HTML guide. If the raw response from curl and the rendered HTML in URL Inspection both lack your content while your browser shows it, the service worker is the likely reason; confirm it by loading the page in a fresh profile. The difference between the two views is explained in raw vs rendered HTML.
How to fix it without giving up the service worker
The goal is simple: every URL's network response should be a complete page on its own. The worker can still make repeat visits faster.
- Render real HTML on the server for every indexable route. Use server-side rendering or static generation so
/products/blue-chairreturns the product's text, title, meta description, canonical tag and links. Keep the app shell approach for logged-in screens that do not need to be crawled. - Switch navigations to network first. Let the worker fetch the full page from the network and fall back to the cache only when the network fails. Visitors still get offline support, and the worker serves the same HTML a crawler sees.
- Keep fragments behind a separate signal. If the worker assembles pages from fragments, serve fragments only for requests that carry the preload header or a dedicated fragment URL, return full pages for everything else, and send
Vary: Service-Worker-Navigation-Preloadon any URL whose response depends on that header. - Keep real status codes on the network. An offline page belongs in the worker, never on the server. Make sure missing pages return 404 and outages return 503 from the network, so crawlers get an accurate signal.
- Version the precache. Change the cache name on each deploy and delete old caches in the
activateevent, as MDN's example does, so returning visitors do not keep reading HTML your crawlers no longer see.
A network-first navigation handler looks like this:
// sw.js: network first for pages, precached offline page only when the network fails
self.addEventListener('fetch', (event) => {
if (event.request.mode === 'navigate') {
event.respondWith(
fetch(event.request).catch(() => caches.match('/offline.html'))
);
}
});After the change, rerun the curl checks on a sample of real URLs from each template, not only the homepage.
Frequently asked questions
Does Googlebot use service workers?
Google's JavaScript documentation does not say. It does document that its renderer does not keep state such as cookies, Local Storage and Session Storage across page loads, so design as if Googlebot always makes a first visit.
Is a service worker bad for SEO?
No. A worker that speeds up repeat visits while the network still returns full HTML does not change what crawlers receive. The problem is a network response that depends on the worker to become a real page.
Can clients.claim() make the worker control the first page load?
No. It lets an activated worker take over pages that are already open, but those pages' HTML was already fetched from the network. It does not change what a crawler receives.
Why does my page look fine in my browser but empty to crawlers?
Your browser has the worker installed and serves the page from its cache. Test with curl, a fresh browser profile or the DevTools Bypass for network option to see the network response.
Will fixing this get my pages cited in AI answers?
No one can promise that. A complete network response removes a technical barrier so crawlers can read the page; whether an assistant uses it depends on many things outside your site.
Check the response a crawler gets
Run the curl commands above on your main templates, then compare with a fresh browser profile. For a quick check without the command line, the free check runs Crawlito's published checks on a single page, with no account needed.
Filed under Rendering and content. Topics: javascript, rendering, caching, raw-html