Skip to content

Cloudflare Pay Per Crawl: how HTTP 402 charging works

How Cloudflare Pay Per Crawl uses HTTP 402 and crawler-price headers, what Allow, Charge and Block do, and what charging means for AI search visibility.

By Crawlito · 7 min read

In short

  • Pay Per Crawl is part of Cloudflare AI Crawl Control and was in closed beta at the time of writing, with Cloudflare acting as merchant of record.
  • A crawler set to Charge that does not offer payment gets HTTP 402 Payment Required with a crawler-price header, and is only served the page once it sends a matching price header signed with Web Bot Auth.
  • You set one default price per zone, at least $0.001 per crawl, and choose Allow, Charge or Block for each crawler; origin responses can override the price per URL with dynamic pricing.
  • Cloudflare warns that setting search engine crawlers to Charge or Block can hurt SEO, and the same logic applies to AI search crawlers that do not pay.
  • Paths such as /robots.txt and /sitemap.xml are always free to crawl, and Configuration Rules can exempt whole sections.
On this page

Cloudflare Pay Per Crawl lets a site owner charge AI crawlers a fixed price for each page they fetch. A crawler you set to Charge that has not agreed to pay gets an HTTP 402 Payment Required response with a crawler-price header instead of the page. If it retries with a payment header and a valid Web Bot Auth signature, Cloudflare serves the page, records the charge and later pays you through Stripe. For AI search, the practical effect is simple: any crawler that does not pay is treated much like a blocked crawler, because it never receives your content.

This post explains the mechanism from Cloudflare's documentation, the three choices you make per crawler, and how to decide without cutting off the AI search crawlers you want. For the other AI settings in the same dashboard, see the Cloudflare AI crawler settings guide.

What Pay Per Crawl is

Cloudflare describes Pay Per Crawl as a feature of AI Crawl Control that lets site owners control and monetize AI crawler access by setting a price per zone. At the time of writing it is in closed beta: you apply through Cloudflare's form, and Enterprise customers can ask their account team.

Three facts frame everything else:

  • Cloudflare is the merchant of record. It handles the charge to the crawler operator and the payout to you, so you do not invoice anyone.
  • It works only for crawlers Cloudflare can identify cryptographically. Crawler operators must follow Web Bot Auth, Cloudflare's verified bots policy, and register their bot with Cloudflare before they can pay.
  • The price is per successful fetch. Cloudflare's FAQ says a crawler is charged each time it accesses a page, including re-crawls of the same page, and only for successful responses. Error responses are not billed.

How HTTP 402 works in Pay Per Crawl

HTTP 402 has existed for a long time without a standard meaning. RFC 9110 still says the code is "reserved for future use". Pay Per Crawl gives it a concrete job, using a few headers Cloudflare defines in its guide for crawler operators.

Step 1: the crawler asks without paying

http
HTTP/2 402
content-type: text/plain; charset=utf-8
crawler-price: USD 0.01

The crawler-price header states the price for that URL.

Step 2: the crawler offers to pay

The crawler sends the request again with one of two headers:

  • crawler-exact-price: the exact price it accepts, which must match crawler-price.
  • crawler-max-price: the most it will pay. If the page costs more, it gets another 402.

The payment header has to be one of the components covered by the crawler's Web Bot Auth signature. An unsigned offer is rejected.

Step 3: the page is served and charged

http
HTTP/2 200
crawler-charged: USD 0.01

The crawler-charged header tells the crawler what it will be billed. If something goes wrong, the response carries a crawler-error header instead. Cloudflare's error code list includes, for example:

crawler-errorStatusMeaning
CrawlerForbidden403The site owner blocked this crawler
MissingCrawlerPrice402No price header was sent
InvalidCrawlerExactPrice402The exact price did not match crawler-price
InvalidCrawlerMaxPrice402The maximum was below the page price
StrongAuthRequired400Valid Web Bot Auth headers are missing
ConflictingPriceHeaders400Both price headers were sent

Allow, Charge or Block: the choice per crawler

Once Pay Per Crawl is enabled for a domain, you set a default price on the Payments tab of AI Crawl Control. Cloudflare's price page gives a minimum of $0.001 USD per crawl. Then, on the Security tab, each crawler gets one of three actions:

ActionWhat the crawler getsUse it for
AllowThe page, freeCrawlers whose visits you want, such as search and user-triggered fetchers
ChargeA 402 until it pays the set priceCrawlers you are willing to serve only for payment
BlockNo accessCrawlers you do not want at all

You cannot set different prices per crawler: Cloudflare's FAQ says there is a single price for all crawlers set to Charge. Blocking a crawler through WAF rules or Bot Management overrides Charge.

Before any of this, an account Administrator or Super Administrator has to make Pay Per Crawl visible for the domain under Manage Account > Settings > Pay Per Crawl, and payouts need a dedicated Stripe Connect account created from the Cloudflare dashboard. Cloudflare's payouts page says an existing Stripe account cannot be reused, payouts are monthly and subject to minimum thresholds, and at the time of writing the accrued balance is not shown in the dashboard.

Free paths and per-URL pricing

Some paths are always free, so crawlers can still read your rules and discover your URLs: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json.

For more control, Cloudflare's advanced configuration page documents two tools:

  1. Exempt sections with a Configuration Rule. Create a Configuration Rule that matches a URI pattern, for example URI Full wildcard https://*example.com/public/*, and choose Disable Pay Per Crawl. Cloudflare suggests this for home and category pages that help discovery, functional pages like login and search, and promotional or archived content.
  2. Price per URL with dynamic pricing. Select Enable dynamic pricing, then return a crawler-price header from your origin or a Worker. Cloudflare adds a request header to every origin request so your code knows the mode in effect:
http
cf-pay-per-crawl: protocol=cloudflare, pricing=in-band

The pricing value is zone-default, in-band or bypass, the last meaning the request is not subject to payment.

What charging means for AI search visibility

This is the part most announcements skip. A crawler set to Charge that does not pay never sees your page. For an AI search crawler, that has the same effect as a block: the operator cannot read the content, so it cannot use that content in its answers.

Cloudflare says as much for search engines. Its crawler selection page warns that setting crawlers in the Search Engine Crawlers category to Block or Charge may hurt your SEO, because search engines may not be able to index your content. The AI search crawlers sit in the same position:

  • OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that ChatGPT search, Claude and Perplexity draw on. If you charge them and they do not pay, they get a 402.
  • User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User act for a person asking a question right now. Charging them means that person's assistant cannot read your page.
  • Training crawlers such as GPTBot and ClaudeBot do not decide search visibility. They are the natural candidates for Charge if you want payment for training use.

Two things are not documented publicly at the time of writing, so plan around them rather than guessing: which crawler operators have agreed to pay, and how a crawler that receives a 402 treats the URL afterwards. The difference between training, search and user crawlers is the best basis for the decision.

A setup that keeps AI search access while asking for payment for training looks like this:

Crawler typeExamplesSuggested action
Search enginesGooglebot, BingbotAllow
AI searchOAI-SearchBot, Claude-SearchBot, PerplexityBotAllow
User-triggeredChatGPT-User, Claude-User, Perplexity-UserAllow
TrainingGPTBot, ClaudeBot, CCBotCharge or Block

Pair it with matching robots.txt groups, as in allow AI search, block AI training, so crawlers that read robots.txt get the same message as the edge.

How to check your Pay Per Crawl setup

A test from your own machine tells you little, because charging applies to crawlers Cloudflare identifies, not to a request that only borrows a crawler's user agent. Check the configuration instead:

  1. In AI Crawl Control > Security, read the Action column for every AI search and user-triggered crawler. Filter by category and confirm none of them is set to Charge or Block unless you meant it.
  2. Check that no WAF rule or Bot Management setting blocks those crawlers, since a block overrides Charge and Allow.
  3. If you exempted sections, confirm the Configuration Rule's URI pattern matches the real URLs, including the www host.
  4. Watch your logs or Cloudflare analytics for 402 responses to crawlers you intended to allow. A 402 to OAI-SearchBot means it is not getting your pages.

Frequently asked questions

Is HTTP 402 an error for search engines?

RFC 9110 reserves 402 for future use and gives it no defined meaning. Treat it as "not served": a crawler that receives it does not get the page content.

Can I charge different crawlers different prices?

No. Cloudflare's FAQ says one price applies to every crawler set to Charge. You can vary the price by URL with dynamic pricing, not by crawler.

Will charging AI crawlers remove my site from ChatGPT or Perplexity?

If their search crawlers are set to Charge and do not pay, they cannot read your pages, which has the same effect as blocking them. Allow the search and user-triggered crawlers if you want those assistants to be able to use your content.

Does Pay Per Crawl block robots.txt and sitemaps?

No. Cloudflare keeps /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json free to crawl.

Check which crawlers can still read your pages

Pay Per Crawl adds another layer that can stop a crawler you meant to allow. After changing it, run the free check on an important page to see whether the main AI crawlers can fetch it. If you manage several zones or want each crawler's access reviewed with evidence, an audit covers the edge settings and robots.txt together.

Filed under CDN and bot protection. Topics: cloudflare, ai-crawlers, bot-protection, web-bot-auth, http-status