fail2ban banning crawlers: apache-badbots and nginx jails
Why fail2ban bans search and AI crawlers, what apache-badbots and nginx-limit-req really match, and how to find, unban and exempt crawler IPs safely.
By Crawlito · 9 min read
In short
- fail2ban bans at the firewall, so a banned crawler gets no HTTP status at all: the stock iptables and nftables actions reject the connection, and a DROP blocktype makes it time out.
- The stock apache-badbots filter matches an old list of email harvesters generated in 2013, not GPTBot, ClaudeBot or other current AI crawlers, but its jail bans on a single match for 48 hours.
- The nginx-limit-req jail bans any IP that nginx logs as limited, so a per-bot rate limit that returns 429 can turn into a 10-minute firewall ban for a search crawler.
- Google treats network timeouts and connection resets like 5xx errors, and says unreachable indexed URLs can be removed from its index within days.
- Exempt crawlers with ignoreip or an ignorecommand that checks the operator's published IP ranges, never by trusting the user agent string.
On this page
fail2ban bans crawlers when one of its jails sees enough matching lines from a crawler's IP address in your logs. The usual culprits are the apache-badbots jail, which bans on a single matching user agent, and rate or error jails such as nginx-limit-req, which ban any IP that nginx has logged as over a limit. Because the ban happens in the firewall, the crawler gets no HTTP status: its connection is refused or times out. To fix it, find the jail and IP with fail2ban-client, check the IP against the operator's published ranges, unban it, and add an exemption.
This post covers what the stock jails actually match, how to trace a ban back to a crawler, and how to exempt crawlers without opening the door to everything that claims to be one. If you have not decided which crawlers you want, start with blocking AI crawlers by user agent and rate limiting AI crawlers; this post assumes those choices are made and fail2ban is quietly overriding them.
How fail2ban turns log lines into firewall bans
A fail2ban jail combines a filter (regular expressions), a log path, three numbers and a ban action. The upstream jail.conf sets these defaults in [DEFAULT]:
| Setting | Default | Meaning |
|---|---|---|
maxretry | 5 | Matches needed before a ban |
findtime | 10m | Window in which those matches must happen |
bantime | 10m | How long the ban lasts |
enabled | false | Jails are off until you enable them |
Individual jails override these. jail.conf also says not to edit it, because a distribution update will probably overwrite it: put changes in jail.local or a .local file under jail.d/.
The ban action decides what the crawler experiences. In upstream fail2ban, the iptables action uses blocktype = REJECT --reject-with icmp-port-unreachable, and the nftables action, which upstream's Debian paths file sets as the default banaction, uses blocktype = reject. Some admins change the blocktype to DROP. Either way, the request never reaches nginx or Apache:
| Ban action | What the crawler sees | What your access log shows |
|---|---|---|
REJECT / reject (stock) | Connection refused or failed at once | Nothing |
DROP | Connection attempt times out | Nothing |
| No ban | An HTTP response | The request and its status |
That is why fail2ban bans are hard to spot. There is no 403 or 429 to grep for, only a gap where the crawler's requests used to be. Google's guide to network and DNS errors says it treats network timeouts and connection resets similarly to 5xx errors, that crawling starts slowing down immediately, and that already indexed URLs that stay unreachable will be removed from its index within days. It also suggests checking firewall settings for an overly broad blocking rule. For how other connection-level failures look to crawlers, see DNS and TLS errors that stop AI crawlers.
What the apache-badbots filter actually matches
The apache-badbots filter is a user agent list applied to Apache-style access log lines. Its own header calls it a list of "known spambots" and asks you to verify that you really intend to block the IPs it catches. The developer notes say the list was fetched from user-agents.org and generated in November 2013.
At the time of writing, the stock list has two parts:
badbots: mostly email harvesters and old scrapers, such asEmailSiphon,EmailWolf 1.00,ExtractorProandAtomic_Email_Hunter/4.0, plus a few old crawlers such asSnapbot/1.0and several Sogou spider strings.badbotscustom:EmailCollector,WebEMailExtrac,TrackBack/1.02,sogou music spiderandJorgee.
It does not contain GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot or bingbot. So if the stock filter is banning a current AI crawler, someone has edited it. Check for a filter.d/apache-badbots.local file or an edited .conf:
grep -n "badbots" /etc/fail2ban/filter.d/apache-badbots.* /etc/fail2ban/jail.local /etc/fail2ban/jail.d/*The jail is what makes edits risky. The stock [apache-badbots] section sets maxretry = 1 and bantime = 48h, so one request with a matching user agent bans the IP for two days. Add bot or crawler to the list and every search crawler that visits once is gone for 48 hours. If you want to refuse AI training crawlers, a narrow user agent rule that returns 403 leaves robots.txt readable and keeps the IP reachable; a fail2ban ban does neither.
On Plesk, the Fail2Ban documentation lists a preconfigured plesk-apache-badbot jail that scans Apache access logs for email grabbers and vulnerability scanners, and describes apache-badbots as a filter you can use in custom jails. Plesk does not publish that jail's user agent list on that page, so check the filter file on your server before you assume what it matches.
Rate and error jails that catch real crawlers
User agent lists are the obvious suspect, but jails that count errors catch more crawlers, because crawlers request many URLs quickly and some of those URLs fail. The stock web jails in jail.conf include:
| Jail | Log read | What counts as a match | Crawler risk |
|---|---|---|---|
nginx-limit-req | nginx error log | limiting requests (and delaying) lines from limit_req | High if you rate limit crawlers |
nginx-botsearch | nginx error log | Not-found errors for paths like wp-login.php, phpmyadmin, cgi-bin, webmail | Low, unless your site links to such paths |
nginx-bad-request | nginx access log | Responses with status 400 | Medium on sites that return 400 for odd URLs |
nginx-forbidden | nginx error log | access forbidden by rule (nginx deny) | Medium if crawlers hit denied paths |
apache-fakegooglebot | Apache access log | Any Googlebot/ user agent, then a DNS check | Depends on DNS |
nginx-limit-req and per-bot budgets
The nginx-limit-req filter matches every zone by default (ngx_limit_req_zones = [^"]+) and takes the IP from the client: field of the error log line. That matters if you use per-bot zones as described in rate limiting AI crawlers. The zone key may be the bot's name, but nginx still logs the client IP, so the jail bans whichever crawler IP happened to send the excess requests. With the jail.conf defaults, a crawler that gets five refusals in ten minutes from one address is then cut off at the firewall for ten minutes, and your polite 429 with Retry-After becomes a refused connection. The status code makes no difference to the jail: nginx's own default limit_req_status is 503, and the refusal is logged the same way.
Which lines are logged depends on two nginx settings. The limit_req module logs refusals at limit_req_log_level, which defaults to error, and logs delays one level lower. The error_log level defaults to error. With both defaults, only refused requests reach the log and the jail. If you lower error_log to warn or notice, delayed requests are logged too, and the filter matches delaying lines as well.
apache-fakegooglebot and DNS failures
This jail matches every request whose user agent contains Googlebot/, with maxretry = 1, and relies on an ignorecommand to spare real Googlebot. The script does a reverse DNS lookup, requires a hostname ending in google.com or googlebot.com, then checks that the hostname resolves back to the same IP. If the lookup fails or times out, the script reports the IP as not Google, so a resolver problem on your server can ban real Googlebot. Google's verification guide lists googlebot.com and google.com hostnames for crawlers, and gae.googleusercontent.com for some user-triggered fetchers, which this script does not accept. That rarely matters here, because none of the user agents on Google's user-triggered fetchers page contains Googlebot/, so the jail never matches them.
How to find which jail banned a crawler
Work from the IP, not the user agent: a ban is recorded against an address.
# Every jail and how many IPs it holds
sudo fail2ban-client status
# Which jails currently ban this IP
sudo fail2ban-client banned 66.249.66.1
# All IPs a jail has banned, with ban times
sudo fail2ban-client get nginx-limit-req banip --with-time
# Ban history from fail2ban's own log
sudo grep -E "NOTICE +\[[^]]+\] Ban " /var/log/fail2ban.log | tail -50These commands are in the client's command list on current fail2ban; older distribution packages may lack some of them, so check fail2ban-client --help. On Plesk, plesk bin ip_ban --banned lists banned IP addresses, as documented in the ip_ban CLI reference.
Then decide whether each banned IP belongs to a crawler you want. The operators publish JSON files of their ranges, each with a prefixes array of ipv4Prefix or ipv6Prefix entries: for example OpenAI's searchbot.json, Google's common-crawlers.json and Bing's bingbot.json. This script checks banned IPs against downloaded copies:
#!/usr/bin/env python3
# usage: check_bans.py ranges/*.json < banned_ips.txt
import ipaddress, json, sys
nets = []
for path in sys.argv[1:]:
with open(path) as f:
for p in json.load(f).get("prefixes", []):
cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
if cidr:
nets.append((ipaddress.ip_network(cidr), path))
for line in sys.stdin:
for token in line.split():
try:
ip = ipaddress.ip_address(token)
except ValueError:
continue
hits = [src for net, src in nets if ip in net]
print(ip, ", ".join(hits) if hits else "not in any list")Feed it the output of fail2ban-client get <jail> banip. AI crawler IP ranges lists where each operator publishes its file, and how to verify a real AI crawler covers reverse DNS checks for operators that use them. An IP that claims to be GPTBot but is in no list is not GPTBot, and fail2ban is doing its job.
How to unban and exempt crawlers safely
Unbanning fixes today. An exemption stops the next ban.
1. Unban the verified crawler IPs.
sudo fail2ban-client set nginx-limit-req unbanip 66.249.66.1 # one jail
sudo fail2ban-client unban 66.249.66.1 # all jails and the databaseOn Plesk, use Tools & Settings > IP Address Banning (Fail2Ban) > Banned IP Addresses and Unban, or plesk bin ip_ban --unban <ip>,<jail>.
2. Add static exemptions with ignoreip. jail.conf documents it as a list of IP addresses, CIDR masks or DNS hosts, separated by spaces or commas. Set it in jail.local, and keep the loopback entries:
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 192.0.2.10Plesk's equivalent is the Trusted IP Addresses tab, or plesk bin ip_ban --add-trusted. Pasting every crawler range by hand goes stale, because the operators change their files.
3. Use an ignorecommand for published ranges. fail2ban runs the command, with the address in place of <ip>, when a filter matches a line from that IP, and ignores the IP if the command exits with status 0 (the bundled apache-fakegooglebot script works the same way). A small script can check cached copies of the JSON files, refreshed by a scheduled job, so a ban never waits on a download:
# /etc/fail2ban/jail.d/crawlers.local
[nginx-limit-req]
enabled = true
ignorecommand = /usr/local/bin/is_listed_crawler <ip>The script exits 0 when the IP is inside a cached prefix and 1 otherwise. Apply it per jail rather than in [DEFAULT] if you still want the SSH jail to ban everything.
4. Narrow the rules that caught the crawler. Limit nginx-limit-req to the zones meant for abusive clients, for example a per-IP zone on login or search URLs, by creating filter.d/nginx-limit-req.local:
[Definition]
ngx_limit_req_zones = perip_login|perip_searchThen the per-bot zones from your crawler budgets still return 429, but they no longer feed a firewall ban. For apache-badbots, keep custom additions to full, specific tokens, and remember that one hit bans for 48 hours.
5. Never exempt by user agent. fail2ban's ignoreregex can skip lines that mention a crawler name, but anyone can send Googlebot or ClaudeBot. Exempt by verified IP and let the user agent decide only what the web server returns. The same principle applies to other server-side protection, such as ModSecurity and the OWASP CRS.
After the change, test the filter against real logs with fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-limit-req.conf to see how many lines still match, and reload with fail2ban-client reload.
Frequently asked questions
Does the stock apache-badbots jail block GPTBot or ClaudeBot?
No. At the time of writing its list is mostly email harvesters and old scrapers from a 2013 list, and it contains no current AI crawler token. If those crawlers are being banned, look for a local edit to the filter or a different jail.
Why do I see no 403 or 429 for a banned crawler?
The ban is a firewall rule, so the request never reaches the web server. The crawler gets a refused or timed-out connection, and your access log shows nothing for that IP until the ban ends.
Does Plesk use fail2ban for web traffic?
Yes, on Linux. Plesk's documentation lists jails including plesk-apache, plesk-apache-badbot, plesk-modsecurity and recidive, with banned and trusted IPs managed under IP Address Banning (Fail2Ban).
Will a short fail2ban ban hurt my search visibility?
A ban of a few minutes on one IP is unlikely to matter on its own, but repeated bans add up. Google says network timeouts and connection resets slow crawling at once, and that indexed URLs that stay unreachable will be removed from its index within days.
Check that crawlers can reach your pages again
After you unban and add exemptions, watch your access log for the crawler IPs returning with 200 responses. The free check fetches one page the way AI crawlers do, with no account needed. If several layers of server protection are involved, an audit reviews crawler access across up to 500 pages with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: bot-protection, nginx, apache, rate-limiting, server-logs