What is Cloudflare's Pay Per Crawl?

It is Cloudflare's attempt to put a payment layer between AI companies and the sites they crawl. Cloudflare announced on 1 July 2025 that it was changing its default to block AI crawlers unless they pay creators for their content, and said it intended to build a marketplace where publishers and AI companies could transact. Coverage since has described it moving towards paying publishers when their content shapes an answer rather than simply when a page is fetched.

Does Cloudflare block AI crawlers by default now?

For a meaningful share of sites, yes. Cloudflare's own 2025 announcement set blocking as the default position. Reporting through 2026 describes the default tightening further from 15 September 2026 to cover crawlers that mix search with training and agent use, on ad-carrying pages, with scope varying by plan and whether the site is new to Cloudflare. That later change rests on secondary reporting rather than a Cloudflare post we have been able to verify, so check your own dashboard rather than trusting any article including this one.

Will blocking AI crawlers protect my content?

It reduces authorised crawling of it, which is not the same thing. Blocking stops well-behaved crawlers that identify themselves and respect robots.txt. It does nothing about content already absorbed into models, about third parties describing you, or about crawlers that ignore the rules. Blocking is a licensing decision, not a security control.

Should I block AI crawlers on my business website?

For most commercial sites, no. If your website exists to win enquiries, blocking the crawlers that feed the surfaces your buyers use means withdrawing from the channel to protect pages whose entire job is to be found. The businesses where blocking genuinely merits debate are the ones whose content is the product, where being summarised for free destroys the sale.

How do I check whether my site is blocking AI crawlers?

Read your own robots.txt first, looking for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider. Then check your CDN, because a Cloudflare or similar default can block at the edge whatever your robots.txt says. Then check your access logs, which are the only place that tells you what actually arrived rather than what you permitted.

Is blocking Google-Extended the same as blocking Google?

No, and this is the distinction worth getting right. Google-Extended is a separate token covering some AI training and grounding use. Blocking it leaves ordinary Googlebot indexing in place, so you keep traditional Search and give up some AI use. Blocking Googlebot removes you from Google Search altogether, which is a different and much more expensive decision.

Can I charge AI companies for crawling my site?

Realistically, not yet, unless you are a publisher at meaningful scale. Pay Per Crawl has been described in press coverage as a closed beta with selected publishers. The economics are built around content that fills gaps in AI systems, which favours large original-reporting operations over business websites. Planning around revenue from it would be premature.

What happens to AI visibility if I block the crawlers?

You lose the route in that you control. You can still be described in AI answers through third-party sources that mention you, directories, reviews and press, because blocking your own site does not stop anyone else writing about you. What you give up is the ability to shape what gets said with your own words, which is the part you can actually influence.

AI Crawler Blocking and Pay Per Crawl: What UK Businesses Should Actually Do

Cloudflare now blocks AI crawlers by default on a large share of the sites it sits in front of, and it’s building a payment layer around crawling. If you have not looked at this since you last edited robots.txt, your site’s position may have changed without you touching anything.

The decision this forces is a commercial one, not a technical one. I’ll get to it, but the facts need separating first, because the reporting on this is messier than the headlines suggest.

What’s confirmed, and what’s only reported

I’d normally skip this section. Here it matters, because I went looking for a primary source and found less than I expected.

Confirmed by Cloudflare. On 1 July 2025, in a post it called Content Independence Day, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators for their content, and said it planned a marketplace where creators and AI companies could come together. The post is candid that the hard parts were unsolved, including how to value content. Its own words: “We don’t know all the answers yet.”

Reported, not confirmed. A second round of coverage dated 1 July 2026, including TechCrunch, describes Cloudflare setting a 15 September 2026 deadline requiring AI firms to separate search crawlers from training and agent crawlers, with defaults then blocking the mixed-use ones on ad-carrying pages. Secondary sources disagree on who it applies to, some saying new customers and free-tier sites while existing paid customers keep current behaviour. I could not verify that change against a Cloudflare post.

Also reported. Pay Per Crawl has been described as a closed beta with selected publishers, and as evolving towards paying publishers when their content shapes an answer rather than per fetch, with Ceramic.ai and You.com named as early partners.

So: the direction of travel is well established and the specifics are not. Check your own Cloudflare dashboard. Don’t take the detail from an article, this one included.

Blocking is a licensing decision wearing a technical costume

This is the part I think gets muddled most often, and it leads businesses to block for reasons blocking doesn’t serve.

Blocking a crawler stops a well-behaved bot that identifies itself and honours robots.txt. That’s it. It doesn’t retrieve content already absorbed into a trained model. It doesn’t stop a competitor, a directory or a trade publication describing you, and those descriptions are what fills the gap you left. It doesn’t stop a crawler that ignores the rules, because robots.txt has always been a request.

If your concern is “I don’t want my content training models for free”, blocking is a reasonable if partial answer and the honest framing is licensing.

If your concern is “I don’t want my content stolen”, blocking is not a control and shouldn’t be sold to you as one.

The actual decision

Strip out the politics and it comes down to what your website is for.

If your site exists to win enquiries, blocking the AI crawlers is almost always the wrong trade. Your pages exist to be found. Withdrawing them from the surfaces buyers are moving to, in order to protect them, defeats the purpose of having written them. You’d be paying a real cost in visibility for a benefit that’s mostly symbolic.

If your content is the product, it’s a genuine argument. Publishers, research businesses, subscription media, anyone whose revenue comes from people reading the thing rather than from people enquiring afterwards. Being summarised accurately and for free is a direct substitution for the sale. That’s a different business and a different answer.

If you’re regulated, there’s a third consideration that gets missed. Blocking doesn’t stop AI describing your firm, it just removes your own words from the pool. We’ve written before about AI giving wrong information about companies, and the pattern is consistent: the less authoritative first-party material available, the more the answer gets assembled from directories, forums and whatever is lying around. For an FCA or SRA regulated firm, that’s a worse outcome than being summarised.

I’m a strong believer that most businesses blocking AI crawlers right now have done it for a reason that blocking doesn’t actually address.

How to check what you’re doing

Three places, in this order. Most businesses check the first and assume it’s the answer.

Your robots.txt. Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider and Applebot-Extended. Note that several of these split training from live retrieval, so blocking one token doesn’t do what blocking the other does.

Your CDN or WAF. This is the one people miss. A Cloudflare default can block at the edge regardless of what your robots.txt permits, so the two can disagree and the edge wins. If your site is behind Cloudflare and you haven’t looked at the bot settings this year, you don’t currently know your own position.

Your access logs. Permission and behaviour are different things. Logs are the only place that tells you which crawlers actually arrived, how often, and what they got. We ran this on our own site and it changed what we thought was happening: the crawl budget picture only became visible once we stopped reading robots.txt as if it were a report.

If you do let them in

Allowing the crawlers is the start of the work, not the end of it. An AI crawler that arrives and finds a page it can’t extract anything from has cost you bandwidth and bought you nothing.

The work that follows is the same work as the rest of GEO. Sections that stand alone, answers before reasoning, headings that name the question, consistent facts about your business in machine-readable form. Our pieces on content chunking and query fan-out cover the mechanics.

One practical note worth more than it sounds: check whether your own site is obfuscating its contact details from non-JavaScript crawlers. Cloudflare’s email protection rewrites addresses into JavaScript, which means a crawler without JavaScript sees nothing useful. It’s a small thing that quietly undermines entity consistency, and it’s on by default.

Where to start

Open your Cloudflare dashboard and find out what your current bot settings actually are. Then read your robots.txt. Then get a week of access logs and see which crawlers turned up.

Decide after that. Most of the opinions I read on this were formed before the person checked any of the three.