Skip to content

CloudflareBrowserRenderingCrawler: what the /crawl bot does

CloudflareBrowserRenderingCrawler/1.0 is the /crawl endpoint of Cloudflare Browser Run, run by Cloudflare customers. How it reads robots.txt and how to block it.

By Crawlito · 8 min read

In short

  • CloudflareBrowserRenderingCrawler/1.0 is the fixed user agent of the /crawl endpoint in Cloudflare Browser Run, formerly Browser Rendering, which crawls sites for Cloudflare customers rather than for Cloudflare itself.
  • The /crawl endpoint follows robots.txt, including crawl-delay, and waits 0.5 seconds between requests to the same domain when no crawl-delay is set.
  • By default a /crawl job declares the purposes search, ai-input and ai-train, so a robots.txt Content-Signal that sets any of them to no rejects the job with a 400 unless the customer narrows its declared purposes.
  • Cloudflare documents bot detection ID 128292352 for the /crawl endpoint, but matching it in a WAF rule needs an Enterprise plan with Bot Management.
  • The /crawl user agent cannot be changed by the customer, while other Browser Run methods send an ordinary Chrome user agent.
On this page

CloudflareBrowserRenderingCrawler/1.0 is the user agent of the /crawl endpoint in Cloudflare Browser Run, the headless browser product Cloudflare used to call Browser Rendering. When it shows up in your logs, a Cloudflare customer has started a crawl job against your site, usually to collect pages for a search index, a RAG system or monitoring. It is not Cloudflare indexing the web for its own purposes, and it has nothing to do with whether your own site uses Cloudflare. It follows robots.txt, so a User-agent: CloudflareBrowserRenderingCrawler group is the simplest way to allow or block it.

If you are still working out which bot a log line belongs to, start with how to identify an unknown bot user agent. This post covers this one crawler in depth: what it is, how it treats robots.txt and Content Signals, and the WAF options Cloudflare documents.

Who is behind CloudflareBrowserRenderingCrawler

Cloudflare's list of its own crawlers splits them into two kinds. Products such as Always Online, health checks and load balancing monitors crawl your site because you enabled them on your zone. Browser Run is listed separately, as a developer product that "can crawl third-party websites on behalf of Cloudflare customers". It does not crawl your site as part of a Cloudflare service you have turned on.

The /crawl endpoint documentation describes how a customer uses it. They send one API request with a starting URL, get back a job ID, and Cloudflare's infrastructure then fetches the starting page, URLs from the sitemap and links found on pages. The customer collects the results as HTML, Markdown or JSON. Cloudflare lists building knowledge bases, training AI systems such as RAG applications, research, summarization and monitoring as common uses.

A few documented limits explain what the traffic looks like:

SettingDefaultMaximum
Pages per job (limit)10100,000
Link depth (depth)100,000100,000
Job run timen/a7 days
Cache reuse (maxAge)86,400 seconds604,800 seconds

The customer chooses render: true, the default, which loads each page in a headless browser and runs its JavaScript, or render: false, a fast HTML fetch without JavaScript that runs on Workers. Cloudflare also documents that if the customer runs several jobs against the same domain, they share one per-domain rate limit.

So a handful of requests can mean a customer trying the default limit of ten pages, while a long run of requests points to a larger job with a raised limit. Neither tells you who the customer is: none of the request headers Cloudflare documents names the customer.

How to recognise Browser Run traffic in your logs

The /crawl endpoint is only one way to use Browser Run. Cloudflare's automatic request headers page shows each method with a different default user agent:

Browser Run methodDefault user agentCustomer can change it
/crawl endpointCloudflareBrowserRenderingCrawler/1.0No
Other Quick Actions (screenshot, PDF, Markdown, scrape and so on)A desktop Chrome string, Chrome/152.0.0.0 at the time of writingYes, with userAgent
Puppeteer, Playwright, CDPThe underlying Chrome's defaultYes

That means only the /crawl endpoint announces itself by name. The other methods look like a normal browser in your user agent field, so you cannot tell them from a visitor by user agent alone.

Cloudflare also attaches headers that the customer cannot remove or override:

  • cf-brapi-request-id: a request ID on Quick Actions requests, which includes /crawl.
  • cf-brapi-devtools: a request ID on Puppeteer, Playwright and CDP sessions.
  • cf-biso-devtools: a flag saying the request came from Cloudflare's rendering infrastructure.
  • Signature-agent, Signature and Signature-Input: a Web Bot Auth signature you can check against the key directory Cloudflare links from the same page.

Most access logs do not record these headers by default. A quick check for the user agent works anywhere:

bash
grep "CloudflareBrowserRenderingCrawler" /var/log/nginx/access.log \
  | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head -20

The requests come from Cloudflare IP ranges, and Cloudflare's FAQ says Browser Run does not offer per-request IP rotation. Because any client can send any user agent, Cloudflare itself says the user agent is not a reliable proof of origin. Treat a match as a claim until the signature or the headers confirm it.

How it handles robots.txt and crawl-delay

Cloudflare states that the /crawl endpoint respects robots.txt directives, including crawl-delay. If your file has no crawl-delay, it waits 0.5 seconds between requests to the same domain. URLs it is told not to crawl are returned to the customer with "status": "disallowed" and the outcome robots_disallowed, so the customer can see that you said no.

Cloudflare's robots.txt page for Browser Run gives the exact groups. To block the /crawl endpoint everywhere while leaving other crawlers alone:

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: *
Allow: /

To let it in but keep it out of some sections, and slow it down:

User-agent: CloudflareBrowserRenderingCrawler
Crawl-delay: 5
Disallow: /admin/
Disallow: /private/
Allow: /

Three details are worth knowing:

  1. It only covers /crawl. A robots.txt group for this token does nothing about other Browser Run methods, which send a Chrome user agent and are single-page fetches started by the customer's code.
  2. The crawler reports what it found. Each job returns a robots array for up to the first 50 origins it met, with the state available, absent or fetch_failed and the HTTP status code that established that state. Cloudflare does not document which responses lead to which state, or how the crawler treats your URLs when the state is fetch_failed, so keep robots.txt fast and returning 200.
  3. It reads your sitemap and revisits it. Cloudflare says Browser Run periodically refetches sitemaps and suggests serving them with Last-Modified or ETag, and says it processes pages in <priority> order and uses <lastmod> to spot changed pages.

Support for crawl-delay varies across AI crawlers, and this one documents it. See crawl-delay for AI bots for who else does.

How Content Signals can stop a crawl job

This is the part most site owners do not expect. The /crawl endpoint also enforces Content Signals found in your robots.txt, and it does so before the crawl starts.

Every job declares purposes in a crawlPurposes field, from search, ai-input and ai-train. The default is all three. If your robots.txt sets any declared purpose to no, Cloudflare rejects the job when it is started with a 400 error and the message Crawl disallowed by Content-Signal directive (purpose or use level). So this file:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

blocks a default /crawl job. It does not block a customer who sends "crawlPurposes": ["search"], because that job no longer declares training. Cloudflare calls these declarations trust-based: you rely on the customer stating its use honestly.

The use level works the same way. A job declares contentUse as reference or full, and full is the default. If your robots.txt says use=reference, a default job is rejected and a reference job is allowed. If it says use=immediate, every job is rejected, because the /crawl endpoint stores what it crawls and immediate means no retention.

Your robots.txt saysDefault job (all purposes, full)Job declaring search only, reference
No Content-Signal lineRunsRuns
ai-train=noRejected with 400Runs
search=noRejected with 400Rejected with 400
use=referenceRejected with 400Runs
use=immediateRejected with 400Rejected with 400

Rejection at the start is not the only outcome Cloudflare documents. Individual records in a job's results can also carry the outcomes crawl_purpose_disallowed or content_use_disallowed, meaning a Content Signal stopped that URL.

If your zone uses Cloudflare's managed robots.txt, which is added when Bot Preference Sync is on and at least one AI policy blocks or disallows, its User-Agent: * group carries Content-signal: search=yes, ai-train=no, use=reference at the time of writing, as described in Cloudflare's AI crawler settings. Going by the /crawl rules above, that is enough to reject a default job. A job only gets through if it leaves ai-train out of crawlPurposes and sets contentUse to reference.

Cloudflare does not document which group's Content-Signal line the /crawl endpoint reads when your file also has a User-agent: CloudflareBrowserRenderingCrawler group. For the syntax and the other operators that read these signals, see Content Signals in robots.txt.

How to block or allow it with WAF rules

robots.txt only asks. Cloudflare's own Browser Run docs say so and point to server-side controls for enforcement. There are two routes, and which one you can use depends on your plan.

Match the bot detection ID (Enterprise)

Browser Run is always classed as bot traffic by Cloudflare, and the headers page lists two bot detection IDs:

MethodBot detection ID
/crawl endpoint128292352
Other Quick Actions, Puppeteer, Playwright, CDP119853733

Cloudflare's FAQ says you need an Enterprise plan for this, because the field belongs to Bot Management. It describes a custom rule with the field Bot Detection ID, operator equals and the ID as the value. In the expression editor that field is cf.bot_management.detection_ids:

any(cf.bot_management.detection_ids[*] eq 128292352)

Use Block to keep /crawl jobs out, or Skip, placed first, to let your own crawl jobs past your bot protection. Cloudflare documents the Skip version for scanning your own zone.

Match the user agent (any plan)

Because the /crawl user agent cannot be changed by the customer, a user agent match catches every genuine /crawl request:

(http.user_agent contains "CloudflareBrowserRenderingCrawler")

With Block, this also stops anyone faking the name, which is harmless. Do not use the same expression with Skip to let traffic in: anyone could send that string and skip your bot protection. The same logic applies off Cloudflare, for example in nginx:

nginx
if ($http_user_agent ~* "CloudflareBrowserRenderingCrawler") {
    return 403;
}

Cloudflare says Browser Run does not bypass CAPTCHAs, Turnstile or other bot protection, and that your Bot Management, WAF and Turnstile rules apply to it as to any visitor. If you block it, the customer sees the result: errored records carry your status code, such as 402 or 403, and your response body, so the customer can read a short plain-text reason if you give one.

Frequently asked questions

Is CloudflareBrowserRenderingCrawler a search engine or AI training crawler?

Neither by itself. It is a tool Cloudflare customers use to crawl sites, and Cloudflare lists uses from RAG and knowledge bases to monitoring. The declared crawlPurposes of each job say what that customer intends, and you never see them directly.

Does blocking it affect my Google rankings or AI search visibility?

No documented link exists. Googlebot, Bingbot, OAI-SearchBot and the other search crawlers use their own user agents, listed in our AI crawler user agents list, and a group for CloudflareBrowserRenderingCrawler does not apply to them.

Why does it appear when my site is not on Cloudflare?

Browser Run crawls third-party sites for Cloudflare customers, so your site does not need to use Cloudflare for it to visit. What decides a visit is a customer starting a job that points at your URLs.

Can I find out which customer is crawling my site?

Not from the request. Cloudflare documents request IDs and signatures that let you confirm the traffic came from Browser Run, but no header that names the customer.

Check what crawlers can read

Blocking or allowing one crawler is easy to get wrong when robots.txt, Content Signals and WAF rules all apply. Run the free check on an important page to see the robots.txt rule and response the main AI crawlers get.

Filed under AI crawlers. Topics: cloudflare, ai-crawlers, robots-txt, content-signals, waf