Cloudflare has introduced a major shift in how AI companies access digital content online. As of July 1, the internet infrastructure company now blocks AI web crawlers by default across its network, which powers nearly 20% of all websites globally. This move, aimed at preventing unauthorized scraping, opens the door to a new AI licensing market where content creators can set access terms and pricing.
Website owners will now have control over which AI crawlers can access their material, including the option to charge “pay-per-crawl” fees. This allows creators to monetize their content directly or negotiate licensing deals with AI developers seeking training data.
Originally created for search engines, web crawlers are now heavily used by AI companies to extract large volumes of text, images, and data from websites, often without permission. This practice has contributed to the “zero-click” trend, where users rely on AI tools instead of visiting original content sources, leading to traffic declines for publishers.
Cloudflare’s CEO Matthew Prince says the initiative is about restoring control to creators and fostering a more equitable digital economy. “Original content is what makes the internet great,” he said, “and it’s essential that creators continue making it. Our goal is to put the power back in the hands of creators.”
The initiative has earned support from more than three dozen intellectual property-focused organizations and companies, including the News/Media Alliance, Associated Press, Conde Nast, Time, and Fortune. Universal Music Group (UMG), the world’s largest music company, also voiced strong support.
Boyd Muir, UMG’s Chief Operating Officer, welcomed the change: “This will help address the indiscriminate, disruptive, and unauthorized scraping of both creative and commercial IP by AI model developers. At UMG, we believe AI should be used ethically and with respect for human creativity and copyright.”
Cloudflare’s tools identify and filter bot traffic, distinguishing between malicious actors and web crawlers. They plan to use the same security systems deployed against cyberattacks like DDoS to block unwanted AI bots.
While industry groups have applauded the move, some AI researchers have raised concerns. Shayne Longpre of MIT’s Media Lab warned that overly broad restrictions could interfere with open research and personal use, which may not compete with or exploit web publishers.
Still, Cloudflare says the change is necessary to preserve a healthy internet ecosystem — one that compensates creators and establishes clearer rules in the growing intersection of AI and intellectual property.
