Blocking CSS and JavaScript in robots.txt: what breaks
Disallowing CSS, JavaScript or API paths in robots.txt can stop Google rendering your page as users see it. See what breaks, what is safe to block and the fix.
By Crawlito · 9 min read
In short
- Google says Search won't render JavaScript from blocked files or on blocked pages, so a Disallow on your script folder changes what Googlebot can build.
- Google's robots.txt documentation allows blocking resource files only when the page is not significantly affected without them.
- A longer, more specific Allow rule beats a shorter Disallow, so you can open /assets/js/ inside a blocked /assets/ folder without opening the rest.
- Each host serves its own robots.txt, so a script loaded from another domain follows that domain's rules, not yours.
- In Vercel and MERJ's December 2024 data, the major AI crawlers did not run JavaScript, so for them the content has to be in the HTML whatever your asset rules say.
On this page
- Why do crawlers fetch your CSS, JavaScript and API files?
- What happens to the rendered page when resources are blocked?
- Which resources are safe to block, and which are not?
- How do you find blocked resources on your site?
- Common culprits: WordPress, Next.js, API paths and third-party scripts
- How do you fix the rules without opening the whole site?
- Does this matter for crawlers that do not render at all?
- Frequently asked questions
- Check what a crawler can load
Blocking CSS, JavaScript or API paths in robots.txt can stop Google from rendering your page the way a visitor sees it. Google says its Search "won't render JavaScript from blocked files or on blocked pages", and Apple says Applebot may not render a page correctly if robots.txt blocks the scripts or stylesheets it needs. Whether that matters for you depends on what the blocked files do: a tracking script is a different case from the bundle that draws your product list.
This post explains why rendering crawlers fetch these files, what breaks when they cannot, which resources are reasonable to block, and how to find and fix the rules.
Why do crawlers fetch your CSS, JavaScript and API files?
Googlebot does not only read your HTML. Google's JavaScript SEO basics describe a queue: pages that return a 200 status are queued for rendering, and a headless, evergreen Chromium then loads the page and runs its JavaScript. Googlebot indexes the rendered HTML and finds links in it. To do that, the renderer has to request the scripts, stylesheets and data calls the page makes, and before each request it checks robots.txt. If a URL is disallowed, Googlebot skips the request.
Apple's crawler documentation says something similar: Applebot may render your content in a browser, and if scripts or stylesheets (it names XHR, JS and CSS) are blocked, it may not render the page correctly.
Two details from Google's documentation and from John Mueller are easy to miss:
- robots.txt is checked per host. Google's robots.txt specification says the rules apply only to the host, protocol and port that serve the file. A script loaded from
cdn.example.netfollows that host's robots.txt, not the one on your main domain. - Resources are fetched during rendering, not crawled as pages. In a reply reported by Search Engine Journal in early November 2026, Mueller said Googlebot does not crawl these files as separate pages. It fetches them while rendering, so blocking them affects how the page is rendered.
What happens to the rendered page when resources are blocked?
Google documents the rule, not a list of symptoms. The rule: blocked JavaScript is not rendered. The result depends on what the script was for.
| Blocked resource | What can go wrong |
|---|---|
| The script that draws your main content | The rendered page can be nearly empty, so Google has less to index than a visitor sees |
| The script that builds navigation or internal links | Links that exist only in the rendered page may not be found, because Googlebot finds links in the rendered HTML |
| A call to a blocked API path | The data never loads, and anything the page builds from it is missing |
| CSS that controls layout or visibility | Googlebot sees an unstyled or differently arranged page, which may hide or reorder content |
| A script that only runs analytics or ads | Usually nothing that affects content, but confirm with a rendered test |
Treat the middle column as a way to reason about your own site, not as documented outcomes. Google's robots.txt introduction puts the test simply: you can block resource files such as unimportant image, script or style files if the page is not significantly affected without them, and you should not block them if their absence makes the page harder for Google's crawler to understand.
None of this promises a ranking change in either direction. It tells you whether the crawler can build the same page you intended.
Which resources are safe to block, and which are not?
Use the page's content as the test. If a visitor would still see the same text, links and structure without the file, blocking it is low risk. If not, leave it open.
Usually fine to block, after you check a rendered test:
- A dedicated script or external domain that loads one self-contained feature. Mueller's example was an age-verification popup: he said blocking the popup's domain, or the specific script that loads it, would work, and he noted that ad scripts are commonly blocked the same way through the provider's robots.txt. The condition is that the page still loads properly when Googlebot cannot fetch it.
- Scripts that only serve analytics, and widgets or embeds that add nothing to the page text, such as a chat bubble. These are not Mueller's examples: confirm them with a rendered test.
Do not block:
- The JavaScript that renders your main content, headings, prices or product details.
- A shared bundle that does many things. Mueller said he would not block a script file that does "a bunch of things", because you may block more than the one feature you meant to.
- CSS needed to lay out or reveal content.
- API endpoints that supply content or links to the page.
- Lazy-loading or tab scripts that insert text. See lazy loading, tabs and accordions.
Blocking something on purpose is a legitimate tool. Blocking it by accident, with a folder-wide rule copied from an old guide, is the usual problem.
How do you find blocked resources on your site?
Check the rules, then check the rendering. The first tells you what is disallowed, the second tells you whether it matters.
Step 1: list what your page loads
List the script and stylesheet URLs in the HTML, then open your browser's Network tab, filter by Fetch/XHR and note the data paths the page calls.
curl -s https://example.com/ | grep -oE '(src|href)="[^"]+\.(js|css)[^"]*"' | sort -uThis only finds files named in the initial HTML. Scripts that load other scripts, and API calls, show up in the Network tab instead.
Step 2: compare them with your Disallow lines
Fetch the file and read every Disallow line that could match those paths.
curl -s https://example.com/robots.txtRemember how matching works. Per Google's specification, path values are case-sensitive, * matches any run of characters, $ anchors the end of the URL, and the most specific (longest) matching rule wins. RFC 9309 says the same about the longest match. For a walk-through of testing rules against specific URLs, see how to test robots.txt for AI crawlers.
Step 3: run a rendered test
Google's URL Inspection tool can test a live URL. Under View tested page, the Screenshot tab shows the rendered page, and the panel also lists the page resources loaded and the JavaScript console output. Google notes that a screenshot differing from your live page can result from resources blocked to Google's inspection tool. Google's own JavaScript troubleshooting guide recommends checking that the rendered HTML has all the content you expect.
Compare the screenshot and the rendered HTML with what you see in a browser. If the main text, links or product data are missing, look at which scripts and calls they depend on. The raw-versus-rendered comparison in raw HTML vs rendered HTML shows how to measure the gap.
Common culprits: WordPress, Next.js, API paths and third-party scripts
WordPress. The default virtual robots.txt, according to the do_robots() reference, disallows the admin area and allows admin-ajax.php. It does not list /wp-includes/ or /wp-content/. If your file blocks those, the rules did not come from that default: check for a plugin, a copied snippet or a physical robots.txt file in the site root. The default admin-ajax.php allow rule is the same pattern you may need elsewhere: a specific Allow inside a blocked folder. The WordPress readiness guide covers the rest of a WordPress setup.
Next.js and build folders. Next.js serves its JavaScript and CSS from the /_next/ path, in the /_next/static/ folder, according to the assetPrefix documentation. A Disallow: /_next/ therefore blocks the code and styles that render the page. Other frameworks use other names (/assets/, /static/, /dist/), so check your own build output.
API paths. A broad Disallow: /api/ is common because those URLs are not pages. If a client-rendered page fetches its content from /api/products, that rule blocks the data the page needs. Apple names XHR explicitly. Treat a blocked data call as a blocked resource for any renderer, and check the result in a rendered test. The better fix is to put key content in the server's HTML so it does not depend on the call.
Third-party scripts. You cannot edit another domain's robots.txt, and its rules decide whether Googlebot can fetch its files. If a script from another host is essential, ask whether the page works without it. If it is optional, there is nothing to fix.
How do you fix the rules without opening the whole site?
Add narrow Allow rules for the files that matter, and keep the Disallow for what you want closed. Under the longest-match rule, the more specific Allow wins.
Before:
User-agent: *
Disallow: /assets/
Disallow: /_next/
Disallow: /api/After:
User-agent: *
Disallow: /assets/
Allow: /assets/css/
Allow: /assets/js/
Allow: /_next/static/
Allow: /api/productsThe Allow: /api/products line is an example: open only the data paths a page needs, not the whole /api/ folder. If the folder holds nothing sensitive and the content depends on it, removing the Disallow is simpler. robots.txt is not access control; RFC 9309 states these rules are not a form of access authorization, so keep private data behind real authentication.
After you change the file:
- Fetch it again and confirm the live version has the new lines.
- Re-run the URL Inspection live test and compare the screenshot.
- Remember caches. Google generally caches robots.txt for up to 24 hours, so allow a day before judging the result.
The robots.txt mistakes guide covers the other ways a file can go wrong, including blocked folders and servers that return the wrong status.
Does this matter for crawlers that do not render at all?
Less, and for a different reason. Vercel and MERJ's December 2024 analysis found that the major AI crawlers it measured, including OpenAI's, Anthropic's and Perplexity's, did not execute JavaScript. It named Gemini, which uses Googlebot's infrastructure, and Applebot as exceptions. At the time, ChatGPT's and Claude's crawlers downloaded script files but did not run them.
So for a crawler that does not render, an Allow on /assets/js/ changes nothing: it reads the HTML your server sends. The reverse is also true. Opening your scripts does not make client-rendered text visible to it. That content has to be in the HTML, which is the point of do AI crawlers run JavaScript?. Operators can change this, so treat the 2024 data as a snapshot.
For Googlebot, Applebot and any other crawler that renders, asset rules matter. For the rest, server-rendered content matters.
Frequently asked questions
Does Google need access to my CSS and JavaScript?
Google needs them to render the page as a visitor sees it, and it will not render JavaScript from blocked files. Its robots.txt documentation permits blocking resources only when the page is not significantly affected without them.
Does Disallow in robots.txt remove a page from Google?
No. Google's documentation says robots.txt is not a mechanism for keeping a page out of Google, and a blocked URL can still appear in results without a description. Use a noindex directive or password protection for that.
Can I block only some JavaScript files?
Yes. Mueller said blocking a dedicated script, such as an age-verification popup, generally works if the page still loads properly without it. He advised against blocking a shared file that does many things.
Is it safe to block /wp-includes/ or /wp-content/ on WordPress?
Neither is in WordPress's default virtual robots.txt. Blocking /wp-content/ also blocks images under /wp-content/uploads/. Remove those lines unless you have checked a rendered test and the page is unaffected.
Check what a crawler can load
The free check reads one page with no account and reports its robots.txt rules for AI crawlers, its robots meta tag and headers, and whether the main text is in the raw HTML. It does not replace Google's URL Inspection for rendering. If you suspect folder-wide rules across a larger site, an audit runs the checks across up to 500 pages with evidence and prioritized fixes.
Filed under Rendering and content. Topics: robots-txt, javascript, rendering, google, troubleshooting