Skip to content

Cloudflare error 1010 and 1020: why crawlers get blocked

Cloudflare error 1020 access denied, 1010 browser signature or 1015 rate limit? See which setting causes each one and how to let legitimate bots through.

By Crawlito · 9 min read

In short

  • Cloudflare documents error 1020 as an HTTP 403 returned when a security feature the site owner configured blocks a request; WAF custom rules and managed rules get the WAF block response instead of a 1020.
  • Error 1010 is a 403 for a banned browser signature, and Browser Integrity Check is the setting Cloudflare tells site owners they can turn off for it.
  • Error 1015 is an HTTP 429 from a rate limiting rule; Cloudflare's advice to site owners is to review the thresholds and, for very short windows, try a longer period.
  • A custom rule with the Skip action can exempt chosen traffic from Browser Integrity Check or rate limiting rules without switching either off for everyone, but it cannot skip Bot Fight Mode.
  • The Security Events log shows which feature and rule blocked a request, and you can search it by Ray ID, IP address or user agent.
On this page

Cloudflare error 1020 means a security feature the site owner configured blocked the request, and Cloudflare returns it with HTTP 403. Error 1010 means access was denied based on the client's browser signature, also with a 403. Error 1015 means a rate limiting rule fired, with a 429. All three come from settings on the site itself, so if you own the site, the fix is in your Cloudflare dashboard.

One correction to a common claim: Cloudflare's own documentation says requests blocked by WAF custom rules or WAF managed rules do not get a 1020. They get the WAF block response instead. That matters when you go hunting for the rule, because the rule behind a 1020 is probably not in your custom rules list.

What do Cloudflare 1xxx errors look like to a crawler?

Cloudflare lists these errors in its 1xxx error reference. The three that show up most in crawler and audit-tool output are below.

ErrorTitle in Cloudflare's docsHTTP statusDocumented cause
1010The owner of this website has banned your access based on your browser's signature403The owner blocked the request based on the client's browser; Browser Integrity Check is the setting named for site owners
1015You are being rate limited429A rate limiting rule set by the site owner
1020Access denied403A security feature with a Block action and no custom response matched the request

Cloudflare says the 1xxx code appears in the HTML body of the response, while the HTTP status is returned in the status header. So a tool that only records the status sees "403" and nothing more. To see the code, read the body:

bash
curl -s -o body.html -D headers.txt -A "MyAuditTool/1.0" https://example.com/
head -n 1 headers.txt
grep -ioE "error( code)?:? ?1[0-9]{3}" body.html | head

The page wording can vary, so the grep is deliberately loose. If the body is a normal page, you are not looking at a 1xxx error. See your page as an AI crawler with curl covers the full set of checks.

Also note the Ray ID on the error page or the cf-ray response header. You will need it below.

Error 1010: the browser signature is banned

The docs describe two causes. The first is that the site owner blocked requests based on the client's web browser. The second applies to hosts on *.pages.dev and *.workers.dev, and to custom domains of a Pages project: a request whose User-Agent Browser Integrity Check treats as a non-browser client gets a 1010. Cloudflare's example is the default Python urllib user agent.

Browser Integrity Check (BIC) is on by default. Cloudflare says it looks for HTTP headers commonly abused by spammers and denies access, and that it challenges visitors with no user agent or a non-standard one. For a crawler, that means a missing, empty or library-default user agent is the first thing to rule out.

Cloudflare's resolution for site owners is that you can turn off Browser Integrity Check on the Security Settings page. The docs do not say which exact setting produced a given 1010 beyond that, so confirm in Security Events (see below) before changing anything.

If the target is a pages.dev or workers.dev hostname, the docs say BIC cannot be turned off there, by you or by Cloudflare support, and those blocks are not recorded in your zone's Security Events or Worker logs. The documented fix is for the client to send a descriptive user agent, such as my-app/1.0. It does not need to look like a browser. For a Worker, serve it on a custom domain or route in your own zone, then skip BIC for the affected paths.

Error 1020: a security feature blocked the request

Cloudflare's 1020 page says a security rule with a Block action matched the request and has no custom response configured. Its examples are API Shield schema validation and JWT validation. It states plainly that WAF custom rules and WAF managed rules do not produce a 1020.

What those WAF rules do produce is the WAF block response. By default that is a 403, and on Pro plans and above you can change the status (400 to 499), content type and body of a rule's block response. So a crawler can get a 403 from a WAF rule without any "1020" text in the body. A 403 with a 1020 points toward a different feature, and Security Events will name it.

The practical consequence: do not assume a custom rule is the culprit just because the status is 403. Look up the event. Googlebot blocked due to access forbidden (403) walks through the same layer-by-layer search for Google's crawler.

Error 1015 and rate limiting rules

Error 1015 is the rate limiting error. Cloudflare says the site owner configured rules that restrict how many requests a visitor can make in a given time period, and exceeding the limit returns 1015 with a 429.

A crawler hits this easily. Fetching many pages from one IP in a few seconds is how most crawlers work. In the rate limiting parameters, a Block action returns 429 by default, and you can set a different code between 400 and 499. Once the rate is reached, the rule applies its action for the configured duration, so a short burst can lock a crawler out for longer than the burst lasted.

For site owners, Cloudflare's guidance is to review your rate limiting thresholds and adjust them. If a rule counts requests over a very short window such as one second, it suggests trying 10 seconds instead. Cloudflare also notes that applying rate limiting rules to verified bots might affect SEO. For the wider question of how to throttle crawlers without shutting out the ones you want, read rate limiting AI crawlers without locking out AI search.

How to find the matching event in Security Events

Cloudflare's 1020 page sends site owners to the Security Events log, and the same log is the right place to investigate 1010 and 1015 on your own zone.

  1. Get the Ray ID, IP address and time of a failing request. For a crawler you run, the Ray ID is in the cf-ray response header. For a third-party crawler, you may have only the IP and user agent from your logs.
  2. Open the Events tab under Security > Analytics in the dashboard. The dashboard layout changes from time to time, so look for the Events tab under Analytics.
  3. Convert the UTC time on the error page to your local time before you search.
  4. Add a filter for the Ray ID or the IP address, or filter on a user agent from the source breakdown. You can also show the User agent column in the events list.
  5. Expand the event. It shows which feature and rule blocked the request, which is the thing you need to change.

Two limits from Cloudflare's docs: Security Events are shown per event rather than per request, and results may be sampled, so a narrow time range returns more events. Cloudflare also says Ray IDs are not guaranteed to be unique, so confirm with the IP and time as well.

If you find nothing, the request may have been blocked by a feature that does not log there. The pages.dev and workers.dev case above is one documented example.

How to exempt verified bots without turning protection off

Turning a feature off for the whole site fixes the crawler and also removes the protection. A narrower option is a custom rule with the Skip action. Cloudflare documents that skip rules let specific requests bypass security features that would otherwise block or challenge them.

What a skip rule can and cannot reach, at the time of writing:

  • It can skip rate limiting rules, Managed Rules and Super Bot Fight Mode rules, using the options for those phases.
  • Under the separate Skip products option, it can skip Browser Integrity Check, User Agent Blocking, Zone Lockdown and Security Level.
  • It cannot skip Bot Fight Mode on the Free plan. Cloudflare says that product cannot be customized or reconfigured with custom rules.

To match the right traffic, use verified bot status rather than a user agent string, because anyone can send a user agent. Cloudflare says a verified bot is one it has confirmed is transparent about who it is and what it does: it identifies itself through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and it obeys robots.txt and keeps reasonable request rates. The cf.client.bot field is true for requests from a known good bot or crawler.

A skip rule for Browser Integrity Check (error 1010) might use this expression, with the action set to Skip and the product Browser Integrity Check selected:

cf.client.bot

For rate limiting (error 1015), use the same expression and choose the skip option for all rate limiting rules instead. If the traffic is your own audit tool rather than a verified bot, match its fixed IP address instead, for example ip.src eq 203.0.113.7. Replace that address with yours, and keep the rule's scope small.

Cloudflare keeps the original verified bot categories usable in custom rules, for example Search Engine Crawler and AI Search, but says it now classifies bots by behavior. Check where each bot you care about sits before you rely on a category. Cloudflare and AI crawlers: the settings that block them covers the AI-specific policies, and WAF challenge pages and AI crawlers shows how to allow verified bots through challenges.

If the block is by country or IP list instead, see geo-blocking and IP allowlists for crawlers.

Under Attack mode: what it adds for crawlers

Under Attack mode runs extra security checks and shows visitors an interstitial page. Cloudflare describes it as one of the last resorts for a zone under a layer 7 DDoS attack, and says it temporarily pauses access to the site. The interstitial needs a browser that supports JavaScript to pass it.

A crawler that cannot complete that interstitial reads the interstitial instead of your page. If it was left on after an incident, your whole site can look empty to crawlers. Cloudflare says it is off by default, so check the zone overview and turn it off when the attack ends. If you need it for part of the site, Cloudflare's example is a configuration rule that adjusts the Security Level for a path such as /admin.

Frequently asked questions

Is every Cloudflare 403 an error 1020?

No. Cloudflare says WAF custom rules and WAF managed rules do not get a 1020. They get the WAF block response, which defaults to a 403 but can be customized. A 1010 is also a 403. Read the response body, then check Security Events.

Will changing robots.txt fix error 1010 or 1020?

No. robots.txt is a set of crawl rules that cooperating crawlers read. These errors come from security features that evaluate the request itself, so the fix is a rule, setting or skip rule in Cloudflare. Fix both if both are blocking.

Should I turn Browser Integrity Check off for the whole site?

You can, and Cloudflare documents how, but it removes the check for every visitor. A skip rule or configuration rule for the specific hostname, path or verified bots keeps the rest of the site covered.

How do I know a request really comes from a verified bot?

Cloudflare verifies bots through Web Bot Auth signatures, published IP lists with a stable user agent, or reverse DNS. In a rule, use the verified bot field rather than the user agent string, which anyone can copy.

Check what a crawler gets from your site

Run the free check on a page that returns an error to see what an AI crawler receives: the status, the robots.txt rules and whether the main text is in the HTML. It runs from Crawlito's servers, so your Cloudflare Security Events stay the evidence for which rule fired. If the blocks come and go across a whole site, an audit checks up to 500 pages and lists the fixes in priority order.

Filed under CDN and bot protection. Topics: cloudflare, bot-protection, waf, http-status, troubleshooting