The traffic generated by artificial intelligence systems is growing faster than human user traffic and is beginning to behave as a new layer of activity on the internet. According to an analysis by Fastly of its global network, AI-related requests increased approximately 30% between January and May 2026, at a rate 6.5 times higher than human traffic growth during the same period.
This data has immediate implications for media, e-commerce, SaaS platforms, API providers, and digital services: it’s no longer enough to think only about human visitors, traditional SEO, and malicious bots. The new web will also be read, queried, summarized, and acted upon by crawlers, fetchers, assistants, and autonomous agents. Some of these machines can generate value; others may consume infrastructure, copy content, or overload origin servers without generating clear revenue.
Fastly’s data reveals a new layer of traffic
Fastly distinguishes between human traffic, AI-powered crawlers, and fetchers. Crawlers systematically browse the web to gather information that may feed indexes, models, or retrieval systems. Fetchers, on the other hand, retrieve real-time data in response to specific user actions via an assistant or agent.
This distinction is important because not all AI bots have the same impact. A crawler may browse thousands of pages without immediate commercial intent. A fetcher might retrieve data precisely when a user asks about a product, compares offers, searches for a return policy, verifies a fact, or requests an agent to complete a task.
| Metrics observed by Fastly | Data published |
|---|---|
| Period analyzed | January 1 to May 31, 2026 |
| AI traffic growth | Approximately +30% |
| Rate compared to human traffic | 6.5 times faster |
| AI traffic in May composed of crawlers | 85% |
| AI traffic in May composed of fetchers | 15% |
| AI requests requiring origin access | Over 51% |
| Human requests requiring origin access | Less than 9% |
| Growth of Claude-related traffic | Over 555% since January |
| Methodology | Fixed client cohort, with exclusions to avoid distortions |
The 51% figure is particularly significant. For human traffic, most requests can be served from cache, CDN, or intermediate layers without connecting to the origin server. In the case of AI traffic, more than half of the requests analyzed by Fastly require contacting the origin. This increases pressure on infrastructure, databases, APIs, and backend systems.
The reason is straightforward: many AI systems seek fresh information—updated inventories, real-time prices, availability, recent news, reviewed documentation, new policies, or live context. For an assistant, caching isn’t always sufficient. If the user asks about something that just changed, the system will try to retrieve the most recent data.
The issue is no longer just blocking bots
Until recently, many organizations approached automated traffic defensively: detecting bots and blocking them. That approach remains necessary to combat abusive scraping, fraud, credential stuffing, spam, form attacks, and API abuse. However, AI complicates the response.
A bot can be harmful. An agent may be a potential customer. A crawler might threaten content or offer visibility in AI-generated responses. A fetcher might consume resources or act as an intermediary between an interested user and a business.
Fastly summarizes this shift with a clear idea: companies will need to decide which machine traffic to accelerate, which to manage, which to challenge, and which to stop. These decisions are no longer solely in the hands of security teams—they impact marketing, business, content, product, infrastructure, and data strategy.
| Type of Traffic | Behavior | Risk | Potential Value |
| Human | Browsing websites or apps, with session and hourly patterns | Traffic spikes, human fraud, abandonment | Direct conversions, subscriptions, purchases, engagement |
| AI Crawler | Broad and ongoing content indexing | Resource consumption, unauthorized content use | Presence in models, assistants, or response systems |
| AI Fetcher | Targeted query tied to user request | More origin access, infrastructure costs | Intent-driven traffic, discovery, recommendations |
| Autonomous Agent | Executes tasks, compares, reserves, or purchases | Doubtful identity, abuse, lack of limits | New sales channel, APIs, and automation |
| Malicious Bot | Repetitive or aggressive actions | Fraud, scraping, saturation, security threats | None if unregulated |
The key is to stop treating all non-human traffic as the same. Media outlets might want to block extensive training with their content but allow fetchers that respond to user queries with links and attribution. An e-commerce site could block price scraping but permit verified agents to check availability or initiate purchases. SaaS companies might block public APIs but create specific plans for agent integrations.
Origin access can become a hidden new cost
The growth of AI traffic doesn’t just change metrics—it also shifts cost structures. If human traffic is mostly served from cache, the marginal cost per page remains controlled. But if fetchers and agents frequently request dynamic data, the origin server becomes central again.
This impacts content architectures, e-commerce platforms, APIs, and live catalog systems. Each request to the origin may involve database reads, microservice calls, availability calculations, permission validations, or inventory checks. As volume surges, costs won’t only be bandwidth—they’ll involve compute resources, databases, observability, security, scaling, and operations.
| Area Affected | Impact of AI Traffic |
| CDN and cache | Need to distinguish cacheable content from fresh data |
| Origin server | More load if fetchers frequently request real-time data |
| APIs | Higher risk of intensive, potentially unmonetized usage |
| Databases | More dynamic queries and pressure on transactional systems |
| Security | Need to differentiate legitimate agents from malicious automation |
| Observability | New metrics for identifying crawlers, fetchers, and agents | Business | Decisions on access control, monetization, visibility, and blocking |
| SEO and content | Discovery may rely more on assistants than traditional search |
For many technical teams, the first step will be measurement: what percentage of traffic comes from agents or crawlers? Which routes are queried? How many requests reach the origin? Which user-agents are correctly identified? Which providers generate significant load? How much traffic converts into human visits, leads, sales, or referrals? Without this visibility, blocking or allowing traffic will be a blind decision.
Media, e-commerce, and APIs face different challenges
The impact of AI traffic varies depending on the business. For digital media, the main concern may be the relationship between content, attribution, and monetization. If assistants read articles and respond without generating traffic, the advertising model weakens. But if a fetcher cites sources and directs qualified visits, it can open a new discovery channel.
In e-commerce, the dilemma is closer to conversion. An agent comparing prices might be annoying if only extracting data, but valuable if it indicates purchase intent. The business must decide which data to expose, how often, under what limits, and whether to require authentication or paid access.
In APIs, the debate becomes more direct. Agents could become heavy consumers of endpoints. Without quotas, pricing, identity, and limits, AI traffic can inflate costs without generating corresponding revenue.
| Organization Type | Key Question Regarding AI Traffic |
| Digital Media | Should I allow assistants to use my content, and under what conditions? |
| E-commerce | How to distinguish price scraping from genuine purchase intent? |
| SaaS | Which agents can access my APIs, and with what limits? |
| Public Sector | What data should be accessible to agents, and what needs protection? |
| Banking & Insurance | How to verify identity, permissions, and compliance in automated requests? |
| Travel & Booking | Do I accept agents checking availability, comparing, and reserving? |
| B2B Platforms | Should I create specific rates or access for machine-to-machine consumption? |
Strategy centers on visibility, context, and accuracy
Fastly recommends three fundamental elements for managing this new landscape: visibility, context, and accuracy. Visibility reveals who is accessing. Context helps assess whether the access carries value or risk. Accuracy enables tailored responses depending on agent type, accessed route, frequency, intent, and impact on the origin.
Applying a one-size-fits-all policy makes little sense. A company might allow crawlers from certain providers while limiting others, require authentication for fetchers, block sensitive routes, serve cached versions for AI traffic, create specialized APIs for agents, or charge for high-volume structured data access.
Decisions should also align with product and business strategies. Blocking all AI traffic may protect content and reduce load, but can also diminish visibility in assistants shaping user discovery. Allowing everything risks losing control and incurring rising costs. An optimal approach involves dynamic rules.
| Possible Decision | When it Makes Sense | Risks of Misapplication |
| Total Block | Sensitive content, clear abuse, or lack of control | Loss of visibility in assistants |
| Allow Selected Crawlers | Presence in AI and discovery strategies | Content use without clear return |
| Limit Fetchers | Protect origin and control costs | Poor responses in agents with genuine intent |
| Require Authentication | APIs, dynamic data, premium services | Barrier to useful agent integrations | Create Endpoints for AI | Structured data, pricing, availability, documentation | Requires governance and maintenance |
| Monetize Access | High volume of automated queries | Commercial friction without perceived value |
Web is no longer built solely for humans
The Fastly data signals a major shift. For two decades, companies optimized their websites for browsers, search engines, mobile devices, and social media. Now, they must also optimize for agents. This doesn’t mean giving unrestricted access, but designing a layered access approach that differentiates humans, bots, crawlers, fetchers, and authorized agents.
This transition could impact traditional SEO. If discovery shifts from Google to ChatGPT, Claude, Gemini, or other assistants, companies need to consider how to appear in these responses, protect their content, and measure ROI. No single metric like organic click-through exists; instead, there will be mentions, snippets, citations, agent actions, and indirect conversions.
Security models will also evolve. Previously, bot management focused on spotting harmful automation. Future models will need to incorporate agent identity, provider reputation, intent-based limits, origin policies, and possibly machine-to-machine billing.
The websites of the next years will no longer be just collections of pages viewed by humans. They will form a network of data, APIs, and services queried by systems acting on behalf of users. In this scenario, AI traffic is no longer an anomaly—it becomes part of overall demand.
The key question isn’t whether to allow or block AI, but what AI to permit, for which cases, under what limits, and with what value model. Those armed with data, rules, and architecture will have an advantage. Treating AI as mere noise can lead to discovering too late that a significant portion of future customers no longer browse directly—they ask an agent instead.
FAQs
How much did AI traffic grow according to Fastly?
Fastly reports that AI-related requests increased approximately 30% between January and May 2026, growing at a rate 6.5 times faster than human traffic.
What portion of AI traffic comprises crawlers and fetchers?
As of May 2026, 85% of AI requests were from crawlers and 15% from fetchers, according to Fastly’s published data.
Why is origin access a concern?
Over 51% of AI requests require access to the origin server, compared to less than 9% of human traffic. This can increase costs, latency, and infrastructure load.
What should companies do?
Measure AI traffic, differentiate crawlers and fetchers, protect origin servers, define rules by agent type, evaluate which access drives business, and block abusive automation.