WP Engine robots.txt: staging rules, Redirect Bots, web rules
Why WP Engine robots.txt blocks all bots on wpengine.com addresses, how Redirect Bots 301s crawlers, and how Web Rules can block or tag AI crawlers.
By Crawlito · 9 min read
In short
- WP Engine automatically applies a robots.txt with User-agent: * and Disallow: / on environment addresses ending in wpengine.com, so a disallow-all file there is expected, not a fault.
- Changes to robots.txt only show on environments that use a custom domain, and on shared accounts only Production can have one, so Staging and Development keep the default.
- Redirect Bots, off by default, 301 redirects well-known crawler user agents that request a URL ending in a number or containing a query argument to the parent page.
- WP Engine's Web Rules Engine can deny a user agent with a Header condition, which returns 403 Forbidden, and rules are read from top to bottom with the first match taking priority.
- A physical robots.txt in the site root overrides any file generated by a plugin or theme on WP Engine, as on most WordPress hosts.
On this page
- The four layers that affect crawlers on WP Engine
- Why robots.txt blocks everything on wpengine.com
- How your own robots.txt applies on the live domain
- What Redirect Bots does to crawler requests
- Blocking or tagging AI crawlers with Web Rules
- A quick audit of a WP Engine site
- Frequently asked questions
- Check what crawlers get from your WP Engine site
If you see a robots.txt that says Disallow: / on an address like example.wpengine.com, that is WP Engine's default, not a mistake: the host blocks all bots on its environment subdomains so a site is not indexed on the wrong domain. On your custom domain, your own robots.txt applies. Two other WP Engine layers can change what crawlers get: Redirect Bots, which 301s known crawlers away from URLs that end in a number or carry a query string, and the Web Rules Engine, where a user-agent rule can return 403 to a crawler or add response headers such as X-Robots-Tag, whatever your robots.txt says.
This post covers what WP Engine documents about each layer and how to check it. For how WordPress builds robots.txt in general, see how to edit robots.txt in WordPress; for other staging settings that reach production, see staging noindex leaks. WP Engine pages were read on 15 November 2026.
The four layers that affect crawlers on WP Engine
| Layer | Where it is set | What a crawler sees | Default |
|---|---|---|---|
| Environment robots.txt | Platform, on *.wpengine.com addresses | Disallow: / for every bot | Always on for those addresses |
| Your robots.txt | Physical file, plugin, or WordPress virtual file | Your rules, on a custom domain | WordPress or plugin output |
| Redirect Bots | User Portal, environment, Utilities | A 301 to the parent page on some URLs | Disabled |
| Web Rules Engine | User Portal, environment, Web Rules | A 403, a redirect, or extra headers | No rules until you add them |
Each layer is checked differently, so work through them in order when a crawler behaves unexpectedly.
Why robots.txt blocks everything on wpengine.com
WP Engine's robots.txt support page says it restricts search engine traffic on any site using an environment address on the wpengine.com domain. The file is applied automatically to every site on such an address and reads:
User-agent: *
Disallow: /The reason WP Engine gives is to restrict traffic to non-live environments and stop a site being indexed on an incorrect domain, which can affect SEO when the site goes live. Every crawler that follows robots.txt treats this group as a full block, including Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
That matters in two situations:
- Previewing before launch. If you run an AI readiness check, a Lighthouse audit or a crawler test against the
wpengine.comaddress, it will report robots.txt as blocking. That result tells you nothing about the live site. - A live site still on the environment address. If visitors reach your site only through
example.wpengine.com, crawlers are blocked for as long as that is true. The fix is to add and point a custom domain, not to edit robots.txt.
robots.txt applies only to the host, protocol and port it is served from, as Google's robots.txt documentation explains. The environment file has no effect on www.example.com, and your custom domain's file has none on the environment address.
Why your edits do not appear on Staging
WP Engine says only environments that use a custom domain show changes made to robots.txt. Premium accounts can add a custom domain to any environment. Shared accounts can add custom domains only to Production, so Staging and Development usually keep the default file whatever you upload.
This is the opposite of the leak described in the staging post: on WP Engine, the environment address protects itself. The risk is copying a test Disallow: / you wrote for Staging into the Production file, which then applies to your real domain.
How your own robots.txt applies on the live domain
On a custom domain, WP Engine follows the usual WordPress order. Its support page says a physical robots.txt in the site root overwrites any file generated by a plugin or theme. You upload it over SFTP or the SSH Gateway, named in lowercase with a .txt extension. WP Engine mentions Yoast SEO and All in One SEO as ways to create and edit the file.
The page also talks about crawl-delay. At the time of writing it says the default crawl-delay for all sites is 10 seconds and that this is in the server's default robots.txt, although the default file it shows contains no such line. Fetch your own file rather than relying on either. Google does not support crawl-delay at all, so it is not a way to manage Googlebot; crawl-delay and AI crawlers covers which crawlers do read it.
Check what each host serves:
# The live domain: should show your production rules
curl -si https://www.example.com/robots.txt
# The environment address: expect the disallow-all default
curl -si https://example.wpengine.com/robots.txtIf the live domain returns Disallow: / under User-agent: *, or under a named AI crawler you want to allow, that is your file, not WP Engine's default. Remember that a crawler named in its own group ignores the * group; robots.txt groups explained shows how that catches people out.
What Redirect Bots does to crawler requests
WP Engine's Redirect Bots page describes a setting that 301 redirects well-known crawler user agents to the parent page when they request a URL that ends in a number or contains a query argument. Its example: with the setting on, bot visits to site.com/page/2019, site.com/page/2020 and site.com/page/2021 are redirected to site.com/page/.
The purpose is load. Crawlers can increment a number in a URL, such as a year, without end, and each new URL misses the cache. Redirecting them protects the server.
Key facts from the page:
- It is disabled by default.
- You toggle it in the User Portal: open the environment, click Utilities, find Redirect Bots, and choose Enable or Disable.
- The redirect is a 301.
- WP Engine does not publish which user agents count as "well-known crawlers". Whether that includes OAI-SearchBot, ClaudeBot or PerplexityBot is not documented.
- WP Engine suggests disabling it if a site-scanning service fails or gets a 301, giving Facebook's URL debugger as an example.
Which URLs it can catch
Read literally, the pattern covers more than year archives. On a typical WordPress site these URL shapes end in a number or carry a query argument:
| URL shape | Example | Matches the documented pattern |
|---|---|---|
| Paginated archive | /blog/page/2 | Ends in a number |
| Date archive | /2025/11 | Ends in a number |
| Plain permalink | /?p=123 | Query argument |
| Filtered shop listing | /shop/?orderby=price | Query argument |
| Tracked link | /pricing/?utm_source=newsletter | Query argument |
For most of these a redirect to the parent is harmless or even helpful. The cost comes when real content lives at such a URL: paginated archives that are the only link path to older posts, product URLs that end in a numeric ID, or sites still using plain ?p= permalinks. A 301 tells a crawler the page has moved permanently, so it may never fetch the content at the numbered URL. HTTP status codes and redirects for AI crawlers explains how crawlers treat a 301.
How to check Redirect Bots
Because the setting is described as matching on user agent, a request that sends a crawler user agent is a reasonable test. Compare it with a browser user agent:
URL="https://www.example.com/blog/page/2"
curl -sI -A "Mozilla/5.0" "$URL" | grep -i -E '^(HTTP|location)'
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL" \
| grep -i -E '^(HTTP|location)'
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" "$URL" \
| grep -i -E '^(HTTP|location)'A 200 for the browser and a 301 with a location pointing at the parent for a crawler means Redirect Bots, or a similar rule, is active. The OAI-SearchBot string is the example OpenAI publishes at the time of writing; its version number changes. If only some crawler names redirect, you have a working list for your site on the day you tested. Since WP Engine does not publish the list, repeat the test after platform changes.
The fix is a choice, not a default. If your important pages do not end in numbers, leaving it on costs little. If they do, turn it off and manage crawl load with a clean sitemap and robots.txt, which WP Engine's page also advises keeping for bots.
Blocking or tagging AI crawlers with Web Rules
The Web Rules Engine lets you self-serve site rules as an alternative to .htaccess rules. You find it on the environment page under Web Rules, sometimes inside the Advanced section, with tabs for Access Rules, Header Rules and Rewrite Rules.
What it can do, from WP Engine's page:
- Access rules allow or deny requests. WP Engine's own example denies a bot by user agent: action Deny, IP All, and a condition of type Header, name
User-Agent, operator "Regex matches", with the bot's name as the value. Denied requests get 403 Forbidden and are not counted as billable visits. - Header rules set or unset response headers. Some headers cannot be changed, including
server,date,x-cacheand any header containing "WPE". - Rewrite rules issue a 301, a 302 or an internal rewrite, with a soft limit of 1,000 per environment.
- Conditions can match URI, query argument, request method, a header such as User-Agent, Host or Referer, GeoIP country, or IP. All conditions on a rule must match.
- Rules are read from top to bottom and the first one read takes priority.
A 403 from an access rule is stronger than robots.txt: the crawler gets no content at all, whether or not it reads robots.txt. Use it for crawlers you never want. If you want to block training but keep AI search, aim the regex narrowly. A pattern such as GPTBot|ClaudeBot|CCBot leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot untouched, while a loose pattern such as bot blocks Googlebot too. WP Engine's page does not say whether the regex match is case-sensitive, so test the rule with each user agent you mean to block and each you mean to allow.
A deny rule with no URI condition, like WP Engine's example, also covers the crawler's request for /robots.txt, so it never sees your rules. Google, for example, treats a 4xx robots.txt response other than 429 as if no file existed; with an access rule that makes no difference, because every page also returns 403. Training vs search vs user crawlers lists which name does what.
The X-Robots-Tag example and why the host condition matters
WP Engine's page includes a header rule to "prevent indexing in the default *.wpengine.com URL": set X-Robots-Tag to noindex, nofollow on all responses, with a Host header condition matching the regex .wpengine.com$. With that condition, the header is added only on the environment address.
If you copy the rule and drop the Host condition, or the regex also matches your live domain, every page on your real site is sent with noindex, nofollow. Check both hosts after adding any header rule:
curl -sI https://www.example.com/ | grep -i '^x-robots-tag'
curl -sI https://example.wpengine.com/ | grep -i '^x-robots-tag'The first command should print nothing; the second should print the header.
A quick audit of a WP Engine site
- Fetch
/robots.txton the live domain, both with and withoutwww, and confirm it shows your production rules. - Expect
Disallow: /on thewpengine.comaddress and ignore tools that flag it there. - In Utilities, note whether Redirect Bots is on, then run the curl test on a paginated or numbered URL.
- In Web Rules, read every access rule with a User-Agent condition and every header rule that sets
X-Robots-Tag, top to bottom. - Fetch a key page with crawler user agents and confirm
200, nox-robots-tag: noindex, and no unexpectedlocation.
Frequently asked questions
Why does my WP Engine robots.txt say Disallow: /?
If you are viewing it on a wpengine.com environment address, that is WP Engine's default, which blocks all bots so the temporary address is not indexed. On your custom domain you should see your own file.
Can I change robots.txt on WP Engine Staging?
Only if the Staging environment has a custom domain. WP Engine says only environments with a custom domain show robots.txt changes, and shared accounts can add custom domains to Production only.
Does WP Engine's Redirect Bots affect AI crawlers?
WP Engine says it applies to well-known crawler user agents but does not publish the list, so it is not documented whether AI crawlers are included. Test with a crawler user agent on a URL that ends in a number.
How do I block an AI crawler on WP Engine without a plugin?
Add a Web Rules access rule with action Deny and a Header condition on User-Agent that matches the crawler's name. Matching requests get a 403, so keep the pattern narrow enough to leave search crawlers alone.
Check what crawlers get from your WP Engine site
The environment default, Redirect Bots and Web Rules all sit outside WordPress, so your dashboard will not show their effect. Run the free check on an important page of your live domain to see its robots.txt rules, status code and X-Robots-Tag as the main AI crawlers would receive them.
Filed under Platform guides. Topics: wordpress, robots-txt, redirects, ai-crawlers, wp-engine