Get out of my website! Cloudflare, one of the world’s largest Internet infrastructure providers, has begun blocking AI web crawlers by default.
Written by Steven Vaughan-Nichols, Senior Contributing Editor
July 1, 2025 at 1:11 p.m. PT
The major Internet Content Delivery Network (CDN), Cloudflare, has declared war on AI companies. Starting July 1, Cloudflare now blocks by default AI web crawlers accessing content from your websites without permission or compensation.
The change addresses a real problem. My own small site, where I track all my stories, Practical Technology, has been slowed dramatically at times by AI crawlers. It’s not just me. Numerous website owners have reported that AI crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, generate massive volumes of automated requests that clog up websites so they’re as slow as sludge. GoogleBot alone reports that the cloud-hosting service Vercel bombards the sites it hosts with over 4.5 billion requests a month.
These AI bots often crawl sites far more aggressively than traditional search engine crawlers. They sometimes revisit the same pages every few hours or even hit sites with hundreds of requests per second. While the AI companies deny that their bots are to blame, the evidence tells a different story.
Thus, on behalf of its two million-plus customers, 20% of the web, Cloudflare now blocks #AI_crawlers. For any new website signing up for its services, AI crawlers will be automatically blocked from accessing its content unless the site owner grants explicit permission. Additionally, Cloudflare promises to detect “shadow” #scrapers — bots that attempt to evade detection — by using behavioral analysis and machine learning. What’s good for the AI goose is good for the gander.
This move reverses the previous status quo, where website owners had to opt out of AI crawling. Now, blocking is the default, and AI vendors must request access and clarify their intentions, whether for model training, search, or other uses, before they’re allowed in.
This change arises not only because of frustrated website owners. Numerous publishing companies, such as The Associated Press, Condé Nast, and ZDNET’s own parent company, Ziff Davis, are frustrated that #AI companies have been “strip mining” the web for content. All too often, this has been done without compensation or consent, and sometimes, ignoring standard protocols like robots.txt that are meant to block #crawlers.
(Disclosure: Ziff Davis, ZDNET’s parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
Moreover, recent court cases have ruled in favor of Meta and Anthropic, finding that their use of copyrighted works was legal under the doctrine of fair use. Needless to say, writers, artists, and publishers don’t like this one bit. Publishers are still worried that the federal government will give AI free rein to do as it wants with their content. AI powerhouses such as #OpenAI and Google are continuing to lobby the government to classify AI training on copyrighted data as fair use.
It’s also worth noting that after the Copyright Office released a pre-publication version of its 108-page #copyright and #AI report, which struck a middle ground by supporting both of these world-class industries that contribute so much to our economic and cultural advancement. However, it added that while some generative AI probably constitutes a “transformative” use, the mass scraping of all data did not qualify as fair use. The next day, the Trump administration fired the head of the Copyright Office and replaced her with an attorney with no prior experience in #copyright_law.
Given all this, it’s no wonder that publishers sought an ally in technology.
As Cloudflare CEO Matthew Prince said in a statement, its new policy is meant to “give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself.”
To complement the move to block AI crawlers, Cloudflare has also launched its “Pay Per Crawl” program. This enables publishers to set their own rates for AI companies that want to scrape their content.
This system is currently in private beta and aims to create a framework where AI firms can pay for access, or be denied if they refuse. Technically, this will be done by dusting off an old, mostly unused web server response, HTTP 402, which responds with a “Payment Required” error message. This means it should be simple to implement and compatible with existing websites and their infrastructure.
Overall, this is a big deal. Thanks to Cloudflare powering such a large portion of the internet, a significant amount of web content could become inaccessible to AI companies unless they negotiate access or pay licensing fees. As Nicholas Thompson, CEO of The Atlantic, noted, “Until now, AI companies have not needed to pay for content licenses because they could simply take it without repercussions. Now they will need to negotiate.”
To this point, most AI companies have been actively against paying for content. As Sir Nick Clegg, former deputy UK Prime Minister and Meta executive, said recently, merely asking artists’ permission before they scrape copyrighted content will “basically kill the AI industry.”
Cloudflare’s new policy is a direct response to this approach and the increasing volume and intrusiveness of AI crawlers that have come with it. It’s also an attempt to stop the siphoning of traffic that would otherwise go to publishers.
Since the rise of AI, traffic to news sites has plunged. For example, Business Insider’s traffic dropped by over half, 55% from April 2022 to April 2025. Left unchecked, Thompson recently predicted that, thanks to AI, the Atlantic staff should expect traffic from Google to drop to zero.
[...]
