October 2026 · Third in our research series, after Lists of bot names miss the agents that matter now and The verifiability gap

Web traffic nearly doubled, and blocking AI crawlers now costs sales

Machines now make more of the web’s traffic than people do. Since September 2025, the shoppers AI assistants send convert better than the average visitor, and an assistant only sends shoppers to sites its crawler can read.

Dozens of stones crowd a stone floor in front of one store’s plate. OpenAI’s and Anthropic’s search crawlers and three agents working for a person each have a path cut to the store; the training crawlers and scrapers around them don’t.
The rule we argue for: search crawlers and agents working for a person get a path in, training crawlers and scrapers don’t.
Traffic
1.8 times
Requests a second on Cloudflare, from 63 million at the end of 2024 to 115 million in September 2026.
Cloudflare
Conversion
plus 90 percent
AI-referred shoppers converted 90% more often than other visitors to US retail sites in September 2026.
Adobe Analytics
Blocking
482
of the 5,577 top-10,000 sites with a readable robots.txt block OpenAI’s search crawler, which keeps them out of ChatGPT search answers.
Sitedex scan, our analysis

1 · Growth

Web traffic nearly doubled, and machines now outnumber people.

Cloudflare says traffic across the sites it serves nearly doubled since the end of 2024. One layer down, ICANN’s registry reports show DNS lookups more than doubled while the number of domains grew 14%. Neither says who is asking, though Verisign’s chief executive puts much of the rise down to AI.

Most of the growth is machines. HUMAN measured automated traffic growing 23.5% in 2025, against 3.1% for people, and Cloudflare says bots passed people in May 2026, more than a year early, pulled forward by AI crawlers and agents. They cost more to serve: Fastly found 51% of AI requests reached a site’s own servers in May 2026, against under 9% of human ones.

The companies building agents expect more. Google, Meta and OpenAI are among 13 applicants for .agent, the most contested name in ICANN’s 2026 round.

DNS lookups more than doubled

Daily DNS lookups across generic top-level domains
Jun 2024390B
Jun 2026856B
Daily lookups across generic domains. ICANN registry reports, compiled by Namedesk. Data

Traffic on Cloudflare nearly doubled

Cloudflare average requests per second
End of 202463M
Sept 2026115M
Average requests a second across the sites Cloudflare serves. Peaks pass 150 million. Data

2 · Conversion

The shoppers AI sends now convert better than the average visitor.

Until August 2025, AI-referred shoppers bought less often than other visitors. In every release since, they’ve bought more often, and the gap reached 90% in September 2026.

In a year, AI-referred shoppers went from −9% to +90%

Above the line, AI-referred shoppers were more likely to buy than other visitors. Below it, less likely.

Conversion lift of AI-referred visits over other visits, by month
Aug ’25−9%
Sep ’25+5%
Oct ’25+16%
Nov–Dec ’25+31%
Mar ’26+42%
May ’26+54%
Jul ’26+60%
Sep ’26+90%
Conversion of AI-referred visits against all other visits to US retail sites, from Adobe’s releases over more than a trillion visits. Bars sit at their real dates; Adobe reported November and December together. Data

These are people who clicked through from an assistant’s answer. The assistant did the research, so they arrive ready to buy. Shopify sees AI visits that land on product pages converting 49% better than organic search, and Meta says more than 10 million people use Muse, its personal agent, with shopping one of the top uses.

Not every source agrees. Alphabet reports Google’s fastest query growth in five years, so search isn’t going away. A peer-reviewed study of 973 stores, using data from before the flip, found ChatGPT visitors converting about 13% below organic search. And Microsoft Clarity found AI assistants sent under 1% of traffic to 1,277 publisher sites. The channel is small, and in Adobe’s retail data it now converts better than the average visit.

3 · Blocking

An assistant only sends shoppers to sites its crawler can read.

OpenAI says sites that block its search crawler, OAI-SearchBot, aren’t shown in ChatGPT search answers, though they can still appear as plain links. In the top 10,000 sites, 482 of the 5,577 robots.txt files we could read block it. 260 name it. The other 222 block every bot, and many of those are CDN and app domains rather than shops.

Most sites that think about AI already treat training and search as two decisions. Of the 932 that block OpenAI’s training crawler by name, 664 leave its search crawler open. For Anthropic it is 634 of 840.

Agents that shop for one person face the same rules. DataDome found 99.3% of Muse requests in its sample came from cloud and CDN networks, where scrapers run too, so a rule aimed at data-center traffic blocks both.

In July we found 598 of 840 sites that block Anthropic’s training crawler have no rule for its agents that answer people, an upper bound since we read declared rules only. In August we found 74% of 38 known AI assistants publish signing keys, so a site can check them instead of guessing from a name.

Most sites that block a training crawler by name leave its search crawler open.

Share of sites blocking a training crawler by name that leave the search crawler open
OpenAI: block GPTBot by name, leave OAI-SearchBot open71%
Anthropic: block ClaudeBot by name, leave Claude-SearchBot open75%
Of the readable robots.txt files in the top 10,000 that block each company’s training crawler by name, the share that leave its search crawler open. Declared rules, not what sites enforce. Data

What it adds up to

Blocking an AI search crawler now costs you its shoppers.

  1. Block OpenAI’s search crawler, and ChatGPT stops showing your pages in its search answers.OpenAI
  2. The shoppers those answers send now convert better than your other visitors, from a small share of visits.Adobe Analytics
  3. Most sites that block AI training by name already leave search crawlers open.Sitedex data, our analysis

The gap has widened in every Adobe release since September 2025. Sites still blocking search crawlers opt out of a channel that now converts better than the average visit.

What to do

Give training and search their own rules.

Sign up freeCheck your robots.txt first
  1. 1

    See every agent that reaches your site

    Sign up free, and DNA shows each AI agent that visits: who it is, who made it, and what it came to do.

  2. 2

    Measure the shoppers AI already sends you

    In your analytics, split out visits referred by chatgpt.com, perplexity.ai and gemini.google.com, and compare their conversion with your average.

  3. 3

    Give training and search their own rules

    Let in search crawlers and agents working for a person, decide what training crawlers get, and stop borrowed names. DNA runs these rules on your site; rules are priced with you.

Data

Download every number.

Each network measures its own traffic, so shares and growth rates differ by source. Adobe’s figures are conversion lifts over non-AI visits, not conversion rates, and count people who clicked from an assistant; they come from Adobe’s September 2026 release and Retail TouchPoints’ coverage of it. The top-10,000 figures are our analysis of Sitedex’s July 3, 2026 scan of declared robots.txt rules (CC BY 4.0), not what sites enforce. The .agent count is from ICANN’s October 7 list; applicants can withdraw with a refund until November 27.

See which AI agents reach your site,and what each came to do.

Sign up free

DNA is free for every site.