Cloudflare: Non-human traffic may reshape publishing models

The rapid growth of AI-driven web traffic is threatening to undermine the economic model that has supported the internet for decades, with publishers facing a future where their content is increasingly consumed by machines rather than humans.

According to Cloudflare, non-human requests now account for almost 60 per cent of web traffic, creating a fundamental challenge for a digital economy built around human attention, page views, advertising and subscriptions.

AI agents do not click advertisements or even necessarily visit websites after consuming their content, meaning publishers are being forced to rethink how they structure, distribute and monetise their information.

For the media industry in particular, the rise of generative AI could have significant consequences as AI engines increasingly answer users' questions directly by synthesising information from multiple sources.

Cloudflare expects this shift to force publishers away from traditional SEO towards Answer Engine Optimisation (AEO), or AI search.

Instead of using established strategies to rank highly in a list of search results, publishers will increasingly need to structure their content so AI assistants recognise and cite them as authoritative sources.

Chief Strategy Officer at Cloudflare, Stephanie Cohen, expects the industry to move towards a permission-based approach to AI access.

"Publishers will increasingly block AI crawlers by default unless there is a licensing agreement or a micro-payment system in place," she said.

Under that model, publishers could demand greater visibility into how their content is being consumed by AI models, alongside mechanisms that allow autonomous AI agents to pay for the information they use.

That could represent a major departure from the traditional web, where publishers have generally relied on advertising, subscriptions and referral traffic to generate revenue.

The challenge is particularly significant because AI companies can consume enormous quantities of online content without necessarily sending the same volume of traffic back to the websites where that information originated.

Cohen also warned that the growth of AI scraping is creating a potentially unnecessary burden on the internet's underlying infrastructure.

Around half of the AI scraping traffic observed across Cloudflare's network hits pages that have not changed since the previous visit, according to the company.

That represents wasted computing resources for both website operators and AI companies, while also increasing energy consumption.

The issue highlights the tension between the rapid expansion of AI and the infrastructure economics underpinning the internet.

Cohen argued that addressing these challenges will require a combination of policy, economics and infrastructure rather than simply technological innovation.

Cloudflare's vision for a "better internet" is therefore both aspirational, and a practical engineering challenge.

Cloudflare sits in front of millions of internet properties and says its network accounts for more than 20 per cent of the web, giving it visibility into the traffic moving between users, websites and increasingly AI systems.

That position could become increasingly important as AI agents move beyond simple question-and-answer interactions and begin autonomously accessing websites, APIs and other digital services.

Is the token model sustainable?

The shift is also raising questions about the sustainability of the token-based pricing model used across much of the AI industry.

For enterprises, unrestricted token consumption can quickly become unpredictable and expensive, particularly as AI systems move from basic chat interfaces to autonomous agents capable of executing multi-step workflows.

Cohen said that the token model can remain viable, but businesses will need greater control over how AI workloads consume resources.

"Organisations need strict budget controls, visibility, and guardrails," she said.

"The token model can work long-term, but only if businesses have the network tooling in place to cap, audit, and manage that spending in real time."

Network-level tooling could increasingly be used to cap, audit and manage AI spending in real time, particularly as organisations deploy multiple agents across different systems and models.

At the infrastructure level, Cohen also sees an opportunity for Cloudflare to position itself between enterprises and the major hyperscalers rather than directly competing with them.

Businesses are increasingly operating across multiple cloud providers, data centres and AI platforms, making connectivity between those environments a more pressing challenge than choosing a single provider.

Enterprise customers are seeking to avoid being locked into a single 'walled garden' as their data, applications and AI workloads become increasingly distributed.

The company describes its role as a neutral connectivity cloud sitting across these environments, providing security and low-latency connectivity while allowing organisations to place workloads wherever it makes the most economic sense.

The strategy reflects a broader shift in enterprise IT, where infrastructure is becoming increasingly distributed and AI is adding another layer of complexity to the movement of data and applications.

For the internet more broadly, the biggest change may be the transition from a web designed primarily for human users to one increasingly populated by autonomous machines.

If AI agents become the dominant consumers of online information, the mechanisms that determine who gets paid, who gets traffic and who controls access to content could look very different from those that defined the first decades of the commercial internet.

It could precipitate a complete overhaul of digital publishing business models, potentially leaving media organisations, already facing significant financial challenges, on the brink of ruin.