The company launches a universal button to prevent the automated tracking of web content by artificial intelligence companies without consent
In a decision that could mark a turning point in the battle for the rights of original content on the Internet, Cloudflare has announced the activation of a new feature: a button that allows users to block all AI bots and crawlers with a single click. The tool is available even for users on the free plan.
The measure responds to the growing unease among publishers, media companies, and digital creators over the massive use of their content to train AI models without permission or compensation. As Matthew Prince, CEO of Cloudflare, stated: “AI crawlers have been scraping content with no limits. Our goal is to give power back to the creators.”
The Value of Data and the Rise of Silent Bots
The rise of generative artificial intelligence has skyrocketed the demand for original content: from articles and essays to user comments. Companies like Google have been known to pay hefty sums for legal access to this data (such as the $60 million annually paid to Reddit), but other companies have chosen less transparent methods.
One of the most controversial cases is that of Perplexity, which has been accused of disguising its bots as regular visitors to bypass access restrictions. In the face of this opacity, Cloudflare has opted for a clear and forceful measure.
How Does This New Blocking System Work?
From now on, any Cloudflare customer can go to the control panel and, in the Security > Bots section, activate the option “AI Scrapers and Crawlers.” Doing so will fortify their website against bots identified as data collectors for AI.
This list will be automatically updated with new signatures as mass scraping behaviors are detected, and the system integrates with Cloudflare’s machine learning algorithms to identify suspicious activities even when bots attempt to disguise themselves.
Overview of the Most Active Bots
According to data collected by Cloudflare in June 2025, the most active bots on its network are:
| AI Bot | Percentage of Sites Accessed |
|---|---|
| Bytespider | 40.40% |
| GPTBot | 35.46% |
| ClaudeBot | 11.17% |
| ImagesiftBot | 8.75% |
| CCBot | 2.14% |
Bytespider, operated by ByteDance (the parent company of TikTok), is among the most aggressive, and GPTBot—managed by OpenAI—does not lag behind. Despite the inclusion of restrictions in robots.txt files, many of these bots ignore such instructions or manipulate their identification.
Pay-per-Scrape: Monetizing Access to Original Content
Alongside the blocking feature, Cloudflare is launching a “Pay-per-Scrape” platform, allowing website owners to set fees for AI companies to access their content legally. For now, this option is only available to some reference media outlets, such as The Atlantic, Stack Overflow, or Quora, but its progressive expansion is expected.
This model aims to restore an ethical and economic balance: if tech companies want to use human content to train their models, they will have to pay for it.
What Does This Mean for the Future of the Internet?
The announcement comes at a pivotal moment. An increasing number of users consult chatbots like ChatGPT directly instead of visiting original websites, reducing traffic to content sites and raising concerns about the sustainability of the digital ecosystem as we know it.
As Prince pointed out: “People have become more trusting of AI in the past six months, which means they aren’t reading original content. That’s a direct threat to creators.”
A Firm Step Toward Transparency and Digital Rights
Cloudflare not only offers protection; it also wants to facilitate the verification of legitimate bots. It is working with AI companies to ensure they declare their purpose (training, inference, search) so that website owners can decide what to allow or block.
With this initiative, Cloudflare reinforces its position as an ally of creators and defenders of intellectual property in the age of artificial intelligence. And it sends a clear warning: the era of scraping without consequences may be coming to an end.
via: blog.cloudflare.com