Skip to content

Cloudflare 520, 521, 522 and 524 errors: what crawlers see

Cloudflare error 522, 520, 521 or 524 in a crawl report means the edge could not get a page from your origin. See each documented cause and what crawlers do.

By Crawlito · 9 min read

In short

  • Cloudflare's 520 to 526 errors are generated at the edge when it cannot get a usable response from your origin, so every visitor and crawler on that request gets a 5xx and no page content.
  • Cloudflare documents a 522 when the origin does not answer a TCP SYN within 19 seconds, or does not acknowledge the request within 90 seconds after connecting, and names blocked or rate-limited Cloudflare IPs as the most common cause.
  • A 524 means Cloudflare connected but the origin sent no HTTP response within the Proxy Read Timeout, which is 125 seconds by default at the time of writing and can be raised only on Enterprise.
  • Google says it ignores content from URLs that return 5xx, slows crawling, and eventually drops URLs that keep failing.
  • A bot rule block looks different: errors 1010 and 1020 arrive as 403, rate limiting error 1015 as 429, and challenge pages carry a cf-mitigated header set to challenge.
On this page

Cloudflare errors 520, 521, 522, 523, 524, 525 and 526 mean Cloudflare's edge tried to fetch the page from your origin server and failed. Error 522 is a connection timeout, 521 is a refused connection, 524 is an origin that connected but never answered, and 520 is an empty or malformed reply. Crawlers get the same 5xx status as people do, so they receive no content, and Google treats repeated 5xx responses as a reason to crawl less and, eventually, to drop the URLs. The fix is almost always at the origin or between Cloudflare and the origin, not in your robots.txt or bot settings.

This post covers the origin side. If your report shows 403s, error 1020 or 1010, or a challenge page instead, see Cloudflare error 1010 and 1020. If the crawler cannot even resolve your hostname or complete TLS with Cloudflare, that is a different failure, covered in DNS and TLS errors for AI crawlers.

What each Cloudflare 52x error means

Cloudflare lists each code on its 5xx error reference, with a separate page per error. The causes below are the ones those pages name.

ErrorCloudflare's titleWhat happened between Cloudflare and the originCauses Cloudflare lists
520Web server returns an unknown errorThe origin replied with something Cloudflare could not useOrigin crash, firewall blocking Cloudflare IPs, headers over 128 KB, empty or malformed response, bad HTTP/2 setup
521Web server is downThe origin refused the connectionWeb server offline, Cloudflare requests blocked or rate limited, nothing listening on port 80 or 443
522Connection timed outThe connection never completed, or the request was never acknowledgedCloudflare IPs blocked or rate limited, overloaded origin, wrong origin IP in DNS, keepalives off, dropped packets
523Origin is unreachableNo network route to the origin's IPWrong A or AAAA record, routing problem, overly broad cloud route table
524A timeout occurredConnected, but no HTTP response in timeLong-running query or overloaded server
525SSL handshake failedTLS between Cloudflare and the origin failedNo valid certificate, port 443 closed, no SNI support, no shared cipher suites
526Invalid SSL certificateThe origin certificate failed validation in Full (strict) modeExpired, self-signed or wrong-name certificate, incomplete chain

525 and 526 concern the hop from Cloudflare to your origin. The certificate a crawler sees on your hostname is Cloudflare's edge certificate, which is why those two errors can appear while a browser shows a valid padlock.

The timeouts Cloudflare documents

Two numbers are worth knowing, both from Cloudflare's pages at the time of writing:

  • 522: Cloudflare's 522 page says the error occurs when Cloudflare does not receive a SYN+ACK within 19 seconds of sending a SYN, or, once connected, does not receive an acknowledgment of its request within 90 seconds.
  • 524: The 524 page gives a default Proxy Read Timeout of 125 seconds, and a Proxy Write Timeout of 30 seconds that cannot be adjusted. Enterprise customers can raise the read timeout up to 6,000 seconds.

A crawler may give up long before either limit. At the time of writing, Google, OpenAI, Anthropic and Perplexity do not publish crawler timeouts in seconds, so a page that takes 60 seconds may never produce a 524 in your Cloudflare analytics and still fail for the crawler. Slow TTFB and crawler timeouts covers that side.

Why your origin firewall can cause 521 and 522

The most useful line in Cloudflare's 522 page is that confirming your host allows all Cloudflare IP ranges addresses the most common cause. The 520 and 521 pages say the same thing in different words: security software or firewalls at the origin may block or rate limit Cloudflare's IPs.

This matters for crawlers in a way that is easy to miss. When a site is proxied, your origin sees Cloudflare's addresses, not the visitor's. Cloudflare's guide to restoring visitor IPs says the origin logs a Cloudflare IP by default and the real address travels in the CF-Connecting-IP header. A host firewall, a rate limiter or a ban tool that counts requests per source IP is therefore counting Cloudflare's edge addresses. A burst of crawler traffic can push one edge address over a limit, and then everything that edge address carries is refused. That is our inference from how proxying works, not a scenario Cloudflare describes, but it explains why crawler-heavy hours and 52x spikes often line up.

Check three places:

  1. The host firewall (iptables, nftables, ufw, a cloud security group). If you restrict the origin to Cloudflare, compare your list against Cloudflare's published ranges; a missing range means some edge locations cannot connect.
  2. Ban and rate-limit tools at the origin such as fail2ban, ModSecurity rules or a security plugin. Make them read the visitor IP from CF-Connecting-IP, and only trust that header from Cloudflare's ranges.
  3. The hosting provider's own protection. Shared hosts sometimes rate limit at their network edge. Cloudflare's general advice is to contact the host first, with the error code, the time and timezone, and the URL.

For nginx, Cloudflare documents the realip module approach:

nginx
# Repeat set_real_ip_from for every range in https://www.cloudflare.com/ips-v4 and /ips-v6
set_real_ip_from 173.245.48.0/20;
real_ip_header CF-Connecting-IP;

With that in place, per-IP limits and logs apply to the real client rather than to Cloudflare. If you then want to limit a specific crawler, do it with a Cloudflare rule rather than an origin ban, so the response is a clear 429 instead of a dropped connection.

What Google does with Cloudflare 5xx responses

Google's page on how HTTP status codes affect its crawlers does not list 520 to 526 by number. It does make category-level statements about 5xx:

  • 5xx and 429 errors make Google's crawlers temporarily slow down.
  • Any content Google receives from a URL that returns 5xx is ignored.
  • Already indexed URLs are kept at first but eventually dropped if the errors persist.
  • Once the server returns 2xx again, Google gradually increases the crawl rate.

Its page on DNS and network errors adds that timeouts and connection resets are handled like 5xx errors, crawling slows immediately, and unreachable indexed URLs can be removed within days. With Cloudflare in front, an origin timeout reaches Googlebot as a 522 or 524 status rather than as a network error, so the 5xx rules are the ones to plan around.

The Search Console Crawl Stats report is where this shows up. Its crawl responses table has a Server error (5XX) row whose chart shows approximately when the errors occurred, and the host status section flags server connectivity problems. A failed robots.txt fetch with a 5xx is more serious: the same help page says Google will slow or stop crawling the site until it gets an acceptable robots.txt response, and that crawling stops for the first 12 hours of failures. If /robots.txt is served by the same failing origin, a 522 storm affects the whole site. For how 5xx appears in the page indexing report, see server error (5xx) in Search Console.

OpenAI, Anthropic and Perplexity do not document how their crawlers treat 5xx responses at the time of writing. What is certain is that a 52x response contains no page content for them to read.

How to tell an origin error from a bot rule block

The first question is whether Cloudflare refused the request on purpose or failed to reach the origin. The status code and headers usually answer it.

What you seeLikely sourceWhere to look
520 to 526Cloudflare could not get a usable response from the originOrigin logs, host firewall, Origin Analytics
403 with a 1xxx code in the bodyA Cloudflare security feature blocked the requestSecurity Events
429, or error 1015A rate limiting ruleSecurity Events, rate limiting rules
Any status with cf-mitigated: challengeA challenge page was servedSecurity Events, bot and WAF settings
500, 502, 503 or 504 with your own error pageYour origin or app answered with an errorApplication logs

Cloudflare's challenge detection page says the cf-mitigated header is set to challenge on every challenge page type, and that challenge is its only valid value. A 52x response is not a challenge, so it will not carry that header.

Then confirm with two requests, one through Cloudflare and one straight to the origin:

bash
# Through Cloudflare, with a crawler user agent
curl -s -o body.html -D - -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://www.example.com/ | grep -iE "^HTTP|^cf-ray|^cf-mitigated"
grep -ioE "error( code)? ?5[0-9]{2}" body.html | head -3

# Straight to the origin, bypassing Cloudflare (203.0.113.10 is your origin IP)
curl -s -o /dev/null -w "%{http_code} %{time_starttransfer}s\n" \
  --resolve www.example.com:443:203.0.113.10 https://www.example.com/

Read the results this way:

  • The edge returns 522 and the direct request also hangs: the origin is down or overloaded.
  • The edge returns 522 but the direct request works: something between Cloudflare and the origin is dropping Cloudflare's traffic. That is usually the firewall or rate limiter described above. If your firewall only admits Cloudflare, the direct test will fail from your laptop; run it on the server itself against 127.0.0.1 instead.
  • The edge returns 520 but the direct request returns 200: Cloudflare's 520 page says an origin 200 served as a 520 points to a malformed response, such as oversized headers or an early connection close. Check header size with curl -sI and look for cookie bloat.

Record the cf-ray value. Cloudflare asks for it when you escalate, and you can filter logs by Ray ID.

How to fix each error and check the fix

ErrorFirst fix to tryHow to confirm
520Read origin error logs for crashes; trim response headers; if the origin's HTTP/2 is broken, turn off HTTP/2 to OriginDirect and proxied requests both return 200
521Start the web server; allow Cloudflare ranges; make sure it listens on 443 for Full or Full (strict)ss -ltn shows the port; proxied request returns 200
522Allow all Cloudflare ranges in the firewall and rate limiters; check the origin IP in Cloudflare DNS; enable keepalives522 count falls in Cloudflare analytics
523Correct the A or AAAA record; fix routes that capture Cloudflare's rangesTraceroute from origin to a Cloudflare IP completes
524Move slow work out of the request; poll long jobs; serve exports from a DNS-only subdomainOrigin response times well under the read timeout
525Install a valid certificate or a Cloudflare Origin CA certificate; open 443; support SNIProxied HTTPS request returns 200
526Fix expiry, hostname or chain; or add the certificate to the Custom Origin Trust StoreFull (strict) works without errors

Cloudflare's error analytics live under HTTP Traffic, where you can filter by edge or origin status code. Cloudflare notes they are based on a 1% sample, so use them for trends, not exact counts.

Two Cloudflare features soften the impact but do not fix the cause. Always Online, available on all plans, only activates when Cloudflare cannot connect to the origin, not when the origin answers with its own 5xx. It then serves a stale copy from Cloudflare's cache or static portions of the page from the Internet Archive, if either exists. Planned downtime is a different case: return a proper 503 with Retry-After, as described in HTTP 503 for maintenance, rather than letting Cloudflare report a 521.

Frequently asked questions

Does a Cloudflare 522 mean Cloudflare is down?

Usually not. Cloudflare's documentation describes 522 as Cloudflare failing to complete a connection to your origin, and names blocked or rate-limited Cloudflare IPs as the most common cause.

Will a 52x error remove my pages from Google?

A short burst should not. Google says it keeps already indexed URLs at first, slows crawling, and drops URLs that persistently return server errors.

Can I fix 52x errors by allowing Googlebot in Cloudflare?

No. Bot and WAF rules decide whether Cloudflare forwards a request; a 52x happens after that, when Cloudflare cannot get a response from the origin.

Why do 522 errors appear mostly when crawlers visit?

Crawler bursts can overload a small origin, and origin tools that rate limit per IP see Cloudflare's addresses rather than the crawler's. Make those tools read CF-Connecting-IP and allow Cloudflare's ranges.

Is a 524 the same as a slow page?

A 524 means the origin sent no response within Cloudflare's read timeout, 125 seconds by default at the time of writing. Crawlers may abandon a page much sooner, so a page can be too slow for them without ever producing a 524.

Check what crawlers get from your pages

52x errors come and go with load, so a single test can miss them. Run the free check on your key pages after a fix to confirm AI crawlers get a 200 and real content. If errors keep returning, an audit reviews the edge settings, origin responses and robots.txt together, with evidence for each finding.

Filed under CDN and bot protection. Topics: cloudflare, http-status, cdn, troubleshooting, google