Cloudflare launches marketplace that lets websites charge AI bots for scraping their sites
A large amount of content from books, papers, websites, etc. is used for training, which is essential for the development of AI models, but most of the time it is used without permission from the authors. For this reason, cloud computing service
Introducing pay per crawl: enabling content owners to charge AI crawlers for access
https://blog.cloudflare.com/introducing-pay-per-crawl/
Content Independence Day: no AI crawl without compensation!
https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/
Cloudflare launches a marketplace that lets websites charge AI bots for scraping | TechCrunch
https://techcrunch.com/2025/07/01/cloudflare-launches-a-marketplace-that-lets-websites-charge-ai-bots-for-scraping/
It was reported that as 'AI summaries' began to appear in Google searches, users who only got information from AI summaries after searching stopped visiting websites, and website traffic plummeted. Cloudflare CEO Matthew Prince commented that this is 'a publisher facing an existential crisis,' pointing out that 'the contract that Google sends traffic in exchange for content is no longer meaningful.'
Cloudflare CEO Matthew Prince warns that publishers are in danger as more people trust AI summaries, causing site traffic to plummet - GIGAZINE
Cloudflare has released several tools for publishers to protect their websites from the rapidly increasing number of AI crawlers from 2024 to 2025. In an interview in 2024 , CEO Prince said, 'These products are laying the foundation for a new type of marketplace where publishers can distribute their content to AI companies and get paid for it.'
As a new marketplace, Cloudflare announced 'Pay per Crawl' on July 1, 2025.
Pay per Crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access. Specifically, domain owners who build sites using Pay per Crawl can set three types of access for their site content for each request: 'Allow crawlers to access content for free,' 'Allow crawlers to read content by setting a fee for reading the entire domain,' or 'Deny access completely with no payment option.'
Below is a diagram that explains how the AI crawler blocking feature works. When an AI crawler requests access to Cloudflare, the request is inspected through AI Audit and Web Application Firewall (WAF). Invalid requests are blocked, and legitimate requests can access website content according to the Pay per Crawl settings.
Cloudflare said about Pay per Crawl, 'Many website operators are forced to choose between leaving their content open for AI to freely use or tightening security and blocking access. Cloudflare started with a simple philosophy: we want content creators to control who has access to their work. After hundreds of conversations with news organizations, publishers, and large social media platforms, we found that they consistently want a third option: 'I want to allow AI crawlers to access my content, but I want to be paid for it.' So instead of a blanket block or free open access, we want to enable content owners to monetize their content at Internet scale.'
Pay per Crawl is a platform that allows you to charge AI crawlers for a new revenue plan, and allows you to charge crawlers that may reduce your site's traffic or block crawlers. In conjunction with the announcement of Pay per Crawl, CEO Prince said, 'Original content is what makes the Internet one of the great inventions of the 20th century. We must work together to protect it. AI crawlers have collected content endlessly. Our goal is to put the power back in the hands of creators while supporting innovation at AI companies,' he said, describing the announcement of Pay per Crawl as 'Content Independence Day.'
Related Posts:
in Web Service, Posted by log1e_dh