Skip to content

WordPress security plugin blocking crawlers: Wordfence checks

Wordfence blocking Googlebot or AI crawlers? Check rate limits, custom blocks, country rules and Live Traffic logs, then fix them without switching protection off.

By Crawlito · 9 min read

In short

  • Wordfence has a separate rate limiting setting for Google's crawlers; its documentation warns that treating Google like any other crawler under a low per-minute limit may temporarily block it.
  • Wordfence documents 503 as its default response when it blocks a visitor, and Google reduces its crawl rate when URLs return 429 or 5xx errors.
  • The Google-specific rate limit option covers Google only; the Wordfence help pages do not say how Bingbot or AI crawlers are classed as bots, so check how they are treated in your own logs.
  • New Wordfence installations log only security-related traffic by default, so Live Traffic will not show normal crawler requests until you switch on full traffic logging.
  • Confirm a blocked 'Googlebot' with a reverse DNS lookup followed by a forward lookup, and do not use 'Anyone claiming to be Google has unlimited access' as a fix.
On this page

If Wordfence or another WordPress security plugin is blocking Googlebot, Bingbot or AI crawlers, the cause can be one of four things: a rate limit set too low, a custom block pattern that is too broad, a country block, or a block entry left over from an old attack. Open Wordfence's block list and Live Traffic first, because they show which rule fired. Then loosen only that rule and keep the firewall on.

This post covers Wordfence because its help pages are public and detailed. Other security plugins have similar features with different names and defaults; I have not checked their documentation. Facts below come from Wordfence's help pages and Google Search Central, read on 11 November 2026. Menus move between plugin versions, so confirm the path in your own dashboard.

How do security plugins end up blocking crawlers?

A crawler can look like an unusual visitor: it may request many pages quickly or hit many missing URLs, and it may arrive from a hosting network. Request rates, 404 counts, hostnames and IP ranges are all things a security plugin's rules can match. Wordfence's Blocking and Rate Limiting pages describe the features that can catch a crawler:

MechanismWhere it lives in WordfenceWhat the crawler can get
Rate limitingFirewall Options, Rate Limiting sectionA temporary block page; Wordfence documents 503 as the default block response
Firewall rulesWeb application firewallA 403 with a "potentially unsafe operation" message
Custom patternsFirewall, Blocking tabA block page, for matching IP ranges, hostnames, user agents or referrers
Country blockingFirewall, Blocking tabA "temporarily limited" message, or a redirect you choose
Old block entriesCurrent blocks listThe same block page, until the entry expires or is removed

Wordfence does not document the status code for country or custom-pattern blocks, so read it from a test request rather than assuming. A 503 tells a crawler to come back later, and Google slows down when many URLs return 429 or 5xx. A 403 tells it the page is refused. Our guide to throttling versus blocking covers the difference.

One more cause sits underneath all of these. Wordfence says that if it cannot read visitor IP addresses correctly behind a proxy or CDN, it can see every visit as coming from the proxy, so one blocked attacker can lock out everyone for the length of the block. Check the "How does Wordfence get IPs" setting if you use Cloudflare or a host proxy.

Rate limiting: crawler and human thresholds

The Rate Limiting section has separate limits for anyone's requests, a crawler's page views, a crawler's 404s, a human's page views and a human's 404s. Wordfence shows a "Very strict" warning at 60 or fewer page views per minute, or 30 or fewer 404s per minute. It says strict values can catch friendly bots, and suggests 120 requests per minute as a sound crawler limit unless bots are overloading your site. It adds that the throttle action is the better choice in most cases, because it slows visitors instead of locking them out for a set time.

The Google option. The setting "How should we treat Google's crawlers" has three choices:

  • Verified Google crawlers have unlimited access. Wordfence checks the visitor with a reverse DNS lookup, so a faked Googlebot header fails.
  • Anyone claiming to be Google has unlimited access. Any visitor with a Googlebot user agent passes, real or not.
  • Treat Google like any other crawler. Wordfence advises against this unless your limits are loose, because a low limit may temporarily block Google.

This setting covers Google only. Wordfence says the crawler limits apply when it detects that a visitor is a bot rather than a human, but the help pages do not say how it makes that call or how Bingbot or AI crawlers are classed. Wordfence's Live Traffic page says logging all traffic adds a request per visitor to better distinguish human from bot traffic, which may improve accuracy when your human and bot limits differ. Check how your own logs show each crawler being treated instead of assuming.

Google's user-triggered fetchers resolve to different hostnames from Googlebot, as the verification section below shows. Wordfence does not say whether its reverse DNS check accepts them, which matters if you see blocked "Google proxy" requests in your logs.

The 404 limits. A crawler that requests many missing URLs can trip the 404 rule: old links after a redesign, or images that no longer exist. Wordfence notes that missing static assets can count, and it lets you list URLs under "Allowlisted 404 URLs" so their 404s are not counted against the crawler throttling rules. Fix the broken URLs where you can instead of raising the limit.

Custom blocks, country blocking and leftover IP ranges

Open the Blocking tab on the Firewall page and read the "Current blocks" list. Each row has a block type of Lockout, IP Block or Advanced Block, plus a detail and a reason. Sort by column to find recent automatic blocks that have not expired.

Look for these:

  • User agent patterns. Wordfence patterns use * as a wildcard and are case insensitive. A pattern such as *bot* would match Googlebot, Bingbot and every AI crawler token. A pattern aimed at one scraper often catches more than intended.
  • Hostname patterns. Wordfence's own examples are *.amazonaws.com and *.linode.com. Wordfence warns that a hostname block can catch legitimate visitors and external services that use the same network, so a block aimed at one bad host can also catch a crawler that connects from it. See geo-blocking and IP allowlists.
  • IP ranges from an old attack. Temporary blocks expire, but a block entered with the IP Address block type is permanent. Type a crawler's full IP into the filter box: Wordfence says this shows whether the address is blocked individually or sits inside a blocked range.
  • Country rules. Wordfence's country blocking page tells you to be careful about blocking North America or European countries because friendly crawlers such as Googlebot are located there. It tells you to keep "Google Search Engine" enabled under "Allowlisted Services", the default. The Firewall Options help page describes "Allowlisted Services" but does not name Google in it, so look for the option in your dashboard.

To remove a block, select the row and click Unblock.

How to read Live Traffic and blocked-request logs

Live Traffic logs at the server level, so it records crawler visits that JavaScript analytics never see. But on new installations Wordfence logs only security-related traffic by default: logins and blocked requests. Normal crawler hits are missing, which can make a healthy crawler look absent. To see everything, set the traffic logging mode to all traffic for a few days. Wordfence recommends the security-only setting on low-cost hosting to reduce server load, so consider switching back afterwards.

Then use the advanced filters:

FilterWhat it shows
User Agent contains GooglebotRequests claiming to be Googlebot
HTTP Response Code match 40*Every 4xx response, including 403 and 429
HTTP Response Code = 503Likely rate limit blocks, if you kept the default block response
Security Event = blocked:wordfenceCustom pattern blocks and similar plugin rules
Security Event = blocked:wafFirewall rule blocks and the Premium Real-Time IP Blocklist

Each IP address has buttons to see its recent traffic and to run a WHOIS lookup. Live Traffic keeps data for a maximum of 30 days. Requests that a CDN blocked never reach WordPress, and a page served from a cache may not run Wordfence at all, so a quiet log does not prove nothing happened. Our guide to AI crawlers in server logs shows how to cross-check with your web server logs.

How to confirm a block hit a real crawler

A user agent is a claim. Anyone can send "Googlebot". Google's verification guide gives two methods. For a manual check, run a reverse DNS lookup on the IP from your log, confirm the host ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP:

bash
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Google says its user-triggered fetchers use google.com or gae.googleusercontent.com hostnames instead of googlebot.com. For automatic checks, Google publishes IP range files such as common-crawlers.json and user-triggered-fetchers.json. If the IP fails both checks, it did not come from a verified Google crawler, and leaving the block in place is reasonable. For OpenAI, Anthropic and Perplexity, see how to verify real AI crawlers.

Safe settings changes, then re-test

Change one thing at a time and write down what you changed.

  1. Read the "Reason" on a block page, or find the blocked request in Live Traffic, so you know which rule fired.
  2. Set the Google option to "Verified Google crawlers have unlimited access", not "Anyone claiming to be Google".
  3. For other crawlers, if the crawler page view limit is below the 120 per minute Wordfence suggests, raise it, and use throttle instead of block.
  4. Keep the 404 rule for scanners. Wordfence describes a low 404 limit with block as an effective way to stop vulnerability scanners. Fix the broken URLs and add known harmless ones to "Allowlisted 404 URLs".
  5. Remove stale entries from "Current blocks". Wordfence's "Allowlisted IP addresses that bypass all rules" option skips every rule for the listed addresses, so use it sparingly, and only once IP detection is correct: Wordfence warns that allowlisting a proxy's ranges while it sees every visitor as the proxy lets attackers bypass protection.
  6. Leave "Enable Rate Limiting and Blocking" on. Turning it off disables all advanced blocking, which removes the protection you installed the plugin for.

To re-test, request a page with a crawler user agent and read the status:

bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/

This runs from your IP, not Google's. Under the Verified setting it is treated as a non-Google visitor, so a 200 here does not prove Googlebot gets through. Use the URL Inspection tool's Test live URL in Search Console as a closer check; Google says it fetches and examines the URL in real time and reports a Page fetch status. It tests access for Google-InspectionTool, not a Googlebot crawl, so confirm in Live Traffic too. Our Googlebot 403 guide shows how to read the result. Once the server returns 2xx again, Google says it gradually raises its crawl rate, and it gives no timeframe.

Why caching plugins can keep showing a lockout page

Wordfence says its block pages carry a "Generated by Wordfence" timestamp. If that time is far from the browser time shown beside it, the page was probably cached, despite headers meant to prevent caching. It names Varnish as one cause and says you may need your host to stop caching those pages.

bash
curl -s https://example.com/ | grep -i "generated by wordfence"

If that returns a block page after you fixed the rule, purge the page cache, the CDN cache and any host cache, then test again. The same caching works in the other direction: Wordfence says cached pages can be served without WordPress or its plugins running, so a country block is not applied to them. A test that passes on a cached URL may not match an uncached one.

Frequently asked questions

Does Wordfence block Googlebot by default?

Wordfence's documentation describes unlimited access for Google as the usual goal and gives a Google-specific option for it. Googlebot can still be blocked by a low limit under "Treat Google like any other crawler", by a custom pattern or by a country rule.

Is "Anyone claiming to be Google has unlimited access" safe?

It stops Google being rate limited, but Wordfence notes that anyone who copies the Googlebot user agent gets the same access. Prefer the Verified option, which uses a reverse DNS check.

Does the Google setting cover AI crawlers like OAI-SearchBot or ClaudeBot?

No. It names Google only. The help pages do not say how other crawlers are classified, so confirm in Live Traffic whether they receive 403 or 503 responses.

How soon will Google recover after I fix a block?

Google says it increases the crawl rate gradually after the server returns 2xx again and gives no timeframe. URLs that keep returning server errors can be dropped from its index, so fix the block before it persists.

Check a page from outside

The free check reads one page the way AI crawlers do, with no account needed. It runs from Crawlito's servers, so a rule based on IP or request rate may treat it differently from Googlebot, and your Live Traffic log and Search Console stay the evidence. If you suspect the same rules catch crawlers across the whole site, an audit covers up to 500 pages with evidence and prioritized fixes. For the other WordPress settings that decide what crawlers see, read WordPress and AI crawlers.

Filed under CDN and bot protection. Topics: wordpress, bot-protection, waf, rate-limiting, troubleshooting, ai-crawlers