Skip to content

Screaming Frog user agent blocked: fix 403s on client audits

Why a Screaming Frog crawl gets 403s or Blocked by Robots.txt, how the SEO Spider picks a robots.txt group, and the allowlist fix that keeps audit evidence honest.

By Crawlito · 8 min read

In short

  • By default the SEO Spider identifies as Screaming Frog SEO Spider and follows robots.txt rules for its own user agent first, then Googlebot, then the * group, obeying only one of them.
  • A URL the SEO Spider skips because of robots.txt shows the status Blocked by Robots.txt with status code 0, not a 403 from the server.
  • Screaming Frog's documented fix for persistent 403s is to allowlist your own IP address and the Screaming Frog SEO Spider user agent in the CDN or host security settings.
  • Screaming Frog notes that many hosts and CDNs block spoofed Googlebot user agents with a 403, so switching to a Googlebot preset is not a reliable workaround.
  • Once your audit has an exception, also record a request made without it, because GPTBot, PerplexityBot and other crawlers outside the exception still get the site's default treatment.
On this page

If your Screaming Frog crawl of a client site returns 403s, the server or a security layer in front of it is refusing the SEO Spider's requests, usually because of its user agent, its request rate or the IP it comes from. If the crawl instead shows Blocked by Robots.txt, nothing was refused: the SEO Spider read robots.txt and chose not to fetch those URLs. The clean fix for the first case is an exception the client controls, for your IP or a custom user agent, and then a second request without the exception so you can report what other crawlers get.

This post covers how the SEO Spider identifies itself and reads robots.txt, how to tell the two failures apart, and how to set up an exception that does not damage the evidence in your audit. Screaming Frog's own pages are the source for its behaviour: the configuration guide, the robots.txt testing tutorial and the HTTP status code tutorial. Menu names below follow those guides at the time of writing and can move between releases, so check them against your own install.

What user agent does Screaming Frog send?

By default the SEO Spider sends its own user agent, which contains the product name Screaming Frog SEO Spider. The full string also carries a version number that changes between releases, so copy the current value from your own install under Configuration > User-Agent before you give it to a client.

The same screen has presets for Googlebot, Bingbot and common browsers, plus a custom option with two separate fields:

FieldWhat it controls
HTTP Request User-AgentThe User-Agent header sent with every request
Robots User-AgentThe name the SEO Spider looks for when it reads robots.txt

The two fields are independent. You can send a browser user agent while still matching robots.txt as Screaming Frog, or the other way round. At the time of writing, Screaming Frog lists changing the user agent as a paid-licence feature.

The SEO Spider also crawls at 5 threads by default, according to its configuration guide, and offers a Limit URI/s setting under Configuration > Speed to cap requests per second. Rate matters for the diagnosis below.

How the SEO Spider picks a robots.txt group

Screaming Frog says the SEO Spider follows only one set of user-agent directives, in this order of priority:

  1. A group for the Screaming Frog SEO Spider user agent, if one exists.
  2. Otherwise, the Googlebot group.
  3. Otherwise, the * group.

If the Robots User-Agent field is left blank, it obeys only the * rules.

The second step is the surprise. RFC 9309 says a crawler with no matching group must obey the * group, but the SEO Spider borrows Googlebot's rules first. So with this file:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

a default Screaming Frog crawl gets through, while AhrefsBot, GPTBot and every other crawler without its own group is shut out. A Screaming Frog crawl that goes well does not show that other crawlers can reach the site. Report robots.txt access for each crawler separately; for SEO tool tokens, see how to block AhrefsBot, SemrushBot and other SEO crawlers.

Respect, ignore, or ignore but report

Under Configuration > Robots.txt, the SEO Spider has three modes:

ModeDownloads robots.txtObeys itReports blocked URLs
Respect robots.txt (default)YesYesYes, as Blocked by Robots.txt
Ignore robots.txt but report statusYesNoYes
Ignore robots.txtNoNoNo

URLs skipped this way appear under Response Codes > Blocked by Robots.txt with status code 0 and a "matched robots.txt line" column showing which rule matched. In list mode, Screaming Frog ticks "ignore robots.txt" automatically, so untick it if you want those statuses.

For a client audit, Ignore robots.txt but report status is usually the honest setting: you crawl everything you have permission to crawl and still record which URLs the client's file disallows. Only use it when the client has agreed you may crawl disallowed paths.

Is it a 403 or a robots.txt block?

The two look similar in a summary but come from different places.

What you seeWhere it comes fromUsual cause
Status code 0, Blocked by Robots.txtThe SEO Spider itselfA Disallow rule in the group it chose
403 on every URL, including the start URLServer, CDN or WAFA user agent rule, bot score or IP reputation
403s that start partway through the crawlServer, CDN or WAFRate limiting or an automatic ban after too many requests
A short "checking your browser" page instead of contentCDN or WAFA challenge, served with a 403 on Cloudflare, a 202 on AWS WAF, or sometimes a 200

A 403 that begins after a few hundred URLs usually points to rate, and Screaming Frog suggests lowering the speed under Configuration > Speed. A 403 from the first request usually points to identity. A challenge page that comes back with a success code is the hardest to spot in a crawl export: check word counts and titles, and read WAF challenge pages and AI crawlers for the headers that give it away.

To confirm which layer answered, repeat one blocked URL with curl and compare headers. Use GET rather than HEAD, because some rules treat them differently (HEAD vs GET for crawler testing):

bash
# Paste the HTTP Request User-Agent value from your own install
UA="Screaming Frog SEO Spider/<version>"
curl -s -o /dev/null -D - -A "$UA" https://client.example/
curl -s -o /dev/null -D - -A "Mozilla/5.0" https://client.example/

If the first returns 403 and the second returns 200 from the same machine, the block keys on the user agent. If both return 403, look at your IP or network.

The clean fix: an exception the client controls

Screaming Frog's guidance runs from quick workarounds to a proper fix: try a browser user agent, lower the crawl speed, and if the 403s continue, add your IP address and the Screaming Frog SEO Spider user agent to the allowlist in the CDN or hosting security settings. The SEO Spider is a desktop application that runs on your machine, so the IP to allowlist is your own machine's or network's, not a Screaming Frog address.

For an agency, the allowlist is the right end state. It is the client's decision, it shows in their configuration, and it can be removed when the audit ends. Ask for something narrow:

  1. Your egress IP, plus a user agent match. Give the client a fixed IP (an office line, a VPN with a static address or a cloud machine) and the exact user agent string from your install.
  2. A rule scoped to the audit. On Cloudflare, that is a custom rule with the Skip action. Cloudflare's skip options cover remaining custom rules, rate limiting rules, managed rules, Super Bot Fight Mode and products such as Browser Integrity Check and Security Level. Cloudflare says Bot Fight Mode, the Free plan's bot feature, cannot be skipped or bypassed with custom rules, though an IP Access rule that matches first stops it from triggering. A client on Bot Fight Mode therefore needs an IP Access rule for your address or must turn the feature off for the audit window.
  3. An end date. Agree when the rule comes out, and put it in the statement of work.

A Cloudflare expression for that rule could look like this:

(ip.src eq 203.0.113.10 and http.user_agent contains "Screaming Frog SEO Spider")

If the client prefers not to tie the rule to an IP, a custom user agent or a custom request header works too. The SEO Spider's Configuration > HTTP Header setting can send any header, so you can agree a unique header name with the client and match on it. Keep its value private and rotate it after the audit, because anyone who learns it gets the same exception.

Some platforms offer a signed route instead. Shopify's signed crawler access, under Online Store > Preferences > Crawler access, creates Web Bot Auth signatures sent as three headers (Signature-Input, Signature and Signature-Agent), and Shopify links to instructions for adding them in Screaming Frog, Sitebulb, Jet Octopus and OnCrawl. A signature lasts at most three months and cannot be renewed.

Why switching to a Googlebot user agent does not help

The Googlebot preset is tempting because many sites allow Googlebot through. Screaming Frog itself warns that many hosts and CDNs block spoofed Googlebot user agents and serve a 403, and suggests a browser user agent instead. Security tools check a claimed Googlebot the way Google documents: either a reverse DNS lookup to a googlebot.com, google.com or googleusercontent.com hostname followed by a forward lookup back to the same IP, or a match against Google's published IP range files. Your laptop passes none of those, and some services also fingerprint the TLS handshake, so a changed header does not change what they see. JA3 and JA4 fingerprints in crawler tests explains that layer, and Cloudflare's Fake Google Bot rule covers one rule that targets it.

Record what a crawler without the exception receives

An allowlisted crawl tells you about the site's pages. It does not tell you what AI crawlers get, and that is often the finding the client most needs. Before or after the crawl, capture a baseline from a machine the exception does not cover:

  • the status code and response headers for the home page and two or three key templates
  • whether the body is the real page or a challenge, with the word count
  • the robots.txt group each crawler you report on would follow

A simulated user agent only tests rules that key on the user agent. Rules based on the real crawler's verified IP or signature cannot be reproduced from your machine, so say so in the report rather than claiming GPTBot or PerplexityBot is blocked or allowed on the strength of a curl test. The agency guide to AI search readiness covers how to present findings like these to clients.

Frequently asked questions

Does Screaming Frog obey robots.txt?

Yes, by default. It follows the group for its own user agent, then Googlebot, then *, and lists skipped URLs as Blocked by Robots.txt with status code 0.

Can I block Screaming Frog with robots.txt?

You can add a group naming the Screaming Frog SEO Spider user agent with Disallow: /, and a default crawl will obey it. The user runs the tool on their own machine and can change the user agent or ignore robots.txt, so it is a request, not a control.

What IP address does Screaming Frog crawl from?

The IP of the machine running the SEO Spider. Screaming Frog says to allowlist your own IP address, which is why a fixed office line or a cloud machine makes the exception easier to keep narrow.

Why does my crawl work but the client says AI crawlers are blocked?

Your crawl may use Googlebot's robots.txt rules, an allowlisted IP or a different user agent. AI crawlers get none of those, so test each one separately and record the result without your exception.

Next step

Once your own crawl is unblocked, check what the main AI crawlers receive on the same pages. The free check tests one page with no account, and a full audit covers up to 500 pages and comes as a PDF with the evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: agencies, audits, waf, bot-protection, robots-txt