Skip to content

ModSecurity and OWASP CRS blocking crawlers: rule 913100

Why ModSecurity rule 913100 or 913102 blocks a crawler, what changed in CRS 4, how to read the audit log and how to write a narrow exclusion instead of disabling it.

By Crawlito · 8 min read

In short

  • CRS rule 913100 matches the User-Agent against scanners-user-agents.data, a hand-curated list of security scanners, and adds a critical anomaly score of 5, which on its own meets the default inbound threshold of 5.
  • CRS 3.3 also shipped rule 913101 for scripting clients such as python-requests and rule 913102 for crawlers such as AhrefsBot, SemrushBot and MJ12bot, both at paranoia level 2; in CRS 4 releases, 913100 is the only detection rule left in that file.
  • At the time of writing, the CRS 4 scanner list matches none of GPTBot, ClaudeBot, PerplexityBot, Googlebot, AhrefsBot or SemrushBot, so a block of those bots on CRS 4 comes from another rule, a host's own rules or another layer.
  • CRS blocks with a 403 by default; a 406 or other status from the firewall means whoever configured it changed the disruptive action.
  • Fix a false positive with a runtime exclusion that removes one rule for a narrow condition, not by setting SecRuleEngine Off.
On this page

ModSecurity blocks a crawler when the OWASP Core Rule Set (CRS) gives the request an anomaly score at or above the blocking threshold. For crawlers and audit tools, the usual trigger is rule 913100, which matches the User-Agent header against a list of security scanner names and adds a critical score of 5. With CRS defaults, that single match is enough for rule 949110 to deny the request with a 403. On CRS 3.3 at paranoia level 2 or higher, rules 913101 (scripting clients) and 913102 (crawlers and SEO bots) can do the same to tools like python-requests, AhrefsBot and SemrushBot.

This post covers what each rule matches, how to read the log entry, why some hosts answer with a 406, and how to exclude one rule for one tool without switching the firewall off. For blocking AI crawlers on purpose with a server rule, see blocking AI crawlers in nginx, Apache and .htaccess. For JavaScript challenges and CAPTCHAs at a CDN, see WAF challenge pages and AI crawlers.

What rule 913100 matches

Rule 913100 lives in REQUEST-913-SCANNER-DETECTION.conf. In the current CRS 4 source it reads, trimmed:

apache
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile scanners-user-agents.data" \
    "id:913100,\
    phase:1,\
    block,\
    capture,\
    msg:'Found User-Agent associated with security scanner',\
    tag:'paranoia-level/1',\
    severity:'CRITICAL',\
    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

Three details explain most surprises:

  • It is a substring match. The ModSecurity reference manual says @pmFromFile "performs a case-insensitive match of the provided phrases", loaded from files with one phrase per line, and that it does not check for word boundaries. Any User-Agent that contains a listed phrase anywhere matches.
  • It runs at paranoia level 1, the default, so it is active on almost every CRS install.
  • It is a list of scanner names, not crawlers. The data file's own header says it is "curated by hand" and describes it as "a baseline of unwanted scanners". Entries include nikto, sqlmap, nmap, nuclei, wpscan, masscan, zgrab, acunetix and Detectify.

That last point matters for AI crawlers. At the time of writing, the CRS 4 data file contains no entry that matches GPTBot, ClaudeBot, PerplexityBot, CCBot, Googlebot, AhrefsBot, SemrushBot or Screaming Frog. If one of those is blocked on a CRS 4 server, look for a different rule ID in the log.

The list does include some commercial tools that site owners run on purpose, such as Detectify, nessus, qualysguard, hexometer and sitelockspider. If you pay for one of those and its requests fail, 913100 is the first suspect.

What CRS 3.3 blocked that CRS 4 does not

CRS 3.3 had three User-Agent rules in the same file, and the two extra ones are the ones that caught crawlers. This table comes from the v3.3.7 and current rule files:

RuleData fileParanoia levelExample entriesIn CRS 4 releases
913100scanners-user-agents.data1nikto, sqlmap, nmap; security scan in 3.3 onlyYes
913101scripting-user-agents.data2python-requests, Python-urllib, libwww-perl, mozilla/4.0 (compatible)No
913102crawlers-user-agents.data2AhrefsBot, SemrushBot, MJ12bot, rogerbot, SearchmetricsBotNo
913110scanners-headers.data1Scanner-specific header namesNo
913120scanners-urls.data1Scanner-specific paths and argumentsNo

The CRS changelog for 4.0.0 records the removal of 913110 and 913120 and the move of "less-used functionality to plugins". It has no separate entry for 913101 and 913102, but the comment above 913100 in the CRS 4 file explains the change: the project "has given up on keeping track of different categories of scanners and scripting agents", partly because attackers change the User-Agent anyway. In the 4.0.0 and 4.30.0 release files and the current source, 913100 is the only detection rule in the 913 file.

Two practical consequences:

  1. An old server can block SEO crawlers and scripts. A host still on CRS 3.3 with the paranoia level raised to 2 blocks AhrefsBot, SemrushBot and MJ12bot through 913102, and any monitoring script that keeps the default python-requests User-Agent through 913101. If your Screaming Frog crawl gets blocked only on one host, check which CRS version that host runs.
  2. Watch for the broad 3.3 phrase. The 3.3 scanner list contains security scan, so any audit tool whose User-Agent includes those words matches 913100 at paranoia level 1.

How anomaly scoring turns one match into a block

In anomaly scoring mode, the CRS default, a matching rule does not block by itself. It adds points, and the blocking evaluation rules compare the total with a threshold: rule 949110 in phase 2 always, and rule 949111 in phase 1 only if you turn on early blocking. The defaults are documented in crs-setup.conf.example:

SettingDefaultEffect
tx.critical_anomaly_score5Points added by a CRITICAL rule such as 913100
tx.inbound_anomaly_score_threshold5Total at which rule 949110 denies the request
tx.blocking_paranoia_level1Highest paranoia level whose rules count towards blocking
Blocking status403What the client receives

The setup file says that at the default threshold "a single critical rule match will cause the request to be blocked". So a crawler that matches 913100 is denied even when nothing else about the request looks suspicious. Rules 913101 and 913102 only add to the score when the paranoia level is 2 or higher.

Why some hosts return 406 instead of 403

CRS uses 403 unless the operator changes the disruptive action. The CRS file RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example says the blocking rules use deny "by default paired with a 'status:403' action", and that you change it with SecRuleUpdateActionById after the CRS rules load. The setup file's self-contained mode example adds that you "can also use a different error status, such as 404, 406". The ModSecurity status action sets that code for deny and redirect. Some hosts set their own status, so a crawler can receive 406 Not Acceptable from a firewall rule, not from content negotiation.

A 406 with an HTML error page, on a URL that returns 200 to a browser, is a strong hint that a WAF denied the request. The error page may not mention ModSecurity at all. If you are not sure which layer answered, identify which WAF blocked the crawler from the response headers first. The cause is the same for every status: the anomaly score reached the threshold.

How to find the rule in the audit log

ModSecurity writes two kinds of evidence. The web server error log gets one line per matching rule, and the audit log gets the full transaction. ModSecurity's recommended configuration sets SecAuditEngine RelevantOnly, logs responses that match SecAuditLogRelevantStatus "^(?:5|4(?!04))", and writes to /var/log/modsec_audit.log. Hosts often move that path. The same file ships with SecRuleEngine DetectionOnly, so on an install that started from it, a server that blocks has had the engine switched to On.

Search for the crawler's User-Agent or the rule IDs:

bash
grep -E '"91310[0-2]"|"94911[01]"' /var/log/modsec_audit.log | tail -n 20
grep -E 'id "91310[0-2]"' /var/log/apache2/error.log | tail -n 20

A block caused by 913100 shows two related messages: the detection, then the evaluation. The exact layout depends on the engine and log format; trimmed and illustrative, they look like this:

[id "913100"] [msg "Found User-Agent associated with security scanner"]
  [data "Matched Data: nikto found within REQUEST_HEADERS:User-Agent: Mozilla/5.00 (Nikto/2.5.0)"]
  [severity "CRITICAL"] [tag "paranoia-level/1"]
[id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"]

The data field is the useful part. The rule builds it from the template Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}, so it shows the text that matched a phrase in the data file and the full header value. That tells you whether the match is a real scanner or a substring inside a legitimate tool's User-Agent. If 949110 shows a total above 5 and 913100 is not present, other rules contributed, and an exclusion for 913100 alone will not help.

To reproduce a 913100 block from your own machine, send a listed phrase:

bash
curl -s -o /dev/null -w '%{http_code}\n' -A 'nikto' https://example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://example.com/

A 403 or 406 for the first request and a 200 for the second confirms that CRS is active and blocking on User-Agent.

How to write a narrow exclusion

The CRS documentation on false positives and tuning calls editing CRS rule files directly "a bad idea" and describes two kinds of exclusion. Configure-time directives such as SecRuleRemoveById go after the CRS include. Runtime exclusions use ctl actions and go before it, typically in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.

Prefer a runtime exclusion tied to something the client cannot fake. User-Agent strings are easy to copy, so an exclusion keyed only on the tool's name lets any scanner through by sending that name. Key it on the tool's published IP addresses instead:

apache
# REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
# Let our paid vulnerability scanner through 913100, only from its own IPs.
SecRule REMOTE_ADDR "@ipMatchFromFile /etc/modsecurity/scanner-ips.txt" \
    "id:1100,\
    phase:1,\
    pass,\
    t:none,\
    nolog,\
    ctl:ruleRemoveById=913100"

The manual says @ipMatchFromFile matches REMOTE_ADDR against IPv4 or IPv6 entries loaded from a file. It also notes that a ruleRemoveById set at runtime must run before the rule it disables, which is why the exclusion belongs before the CRS include. Pick an ID that is not used anywhere else in your configuration. If your server sits behind a CDN or proxy, make sure REMOTE_ADDR holds the client's address, not the proxy's, before you rely on it.

For the CRS 3.3 crawler rule, a configure-time removal is reasonable if you do not want User-Agent-based crawler blocking at all:

apache
# After the CRS include
SecRuleRemoveById 913102

That rule only checks a header any client can set, so removing it gives up little protection. To control which crawlers may visit, use robots.txt, and verify real crawlers by IP or reverse DNS where it matters.

Two things to avoid:

  • SecRuleEngine Off or DetectionOnly for the whole site. The manual says DetectionOnly processes rules "but never executes any disruptive actions". That removes every CRS protection, not just the false positive.
  • Raising the threshold to get one tool through. The CRS setup file warns that going beyond a threshold of 5 for requests disables "entire parts of the WAF, such as LFI, RFI, RCE, and data-exfiltration protections".

On managed hosting you usually cannot edit these files. Send the host the rule ID, the data field and the timestamp from the log, and ask for an exclusion of that one rule for your scanner's IPs.

Frequently asked questions

Does OWASP CRS block GPTBot or ClaudeBot?

Not through 913100 at the time of writing: the CRS 4 scanner list has no entry that matches those User-Agents. A block of an AI crawler on a CRS server more often comes from a host-specific rule, a different CRS rule, or a separate layer such as a CDN or fail2ban.

What is the difference between rules 913100 and 913102?

913100 matches security scanners at paranoia level 1 and exists in CRS 3 and CRS 4. 913102 matched crawlers and SEO bots such as AhrefsBot and SemrushBot at paranoia level 2, and is not part of CRS 4 releases.

Why does my host return 406 when ModSecurity blocks a request?

CRS uses 403 by default, but whoever configures ModSecurity can set a different status on the deny action. A 406 from a firewall means the same thing as a 403: the anomaly score reached the threshold.

Can I whitelist a crawler by User-Agent in ModSecurity?

You can, but any client can then send that User-Agent and skip the rule. Tie the exclusion to the tool's IP addresses with @ipMatchFromFile, and remove only the rule that fires.

Check what your server returns to crawlers

After an exclusion goes live, rerun the curl tests and watch the audit log for the rule ID you excluded. To confirm the main AI crawlers get a normal response from an important page, run the free check, which needs no account. If several firewall layers sit in front of the site, an audit reviews crawler access across up to 500 pages with the evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: bot-protection, waf, apache, nginx, troubleshooting