August 2026 · Research from the D3 bot registry, 2,175 known bots

The verifiability gap

Most known bots can’t prove who they are. We counted what 2,175 of them publish for a site to check.

of known bots publish nothing a site can check: no signing keys and no address ranges. All a site gets is the name in the User-Agent header.2,083 of 2,175
95.8%
advertise a Web Bot Auth key directory, so a site can check their signature on the first request, from any address.79 of 2,175
3.6%
come from a company that publishes its address ranges in a file a machine can read: Google, Microsoft, DuckDuckGo, Apple, OpenAI and Perplexity. That list is everything an IP allowlist can check.13 of 2,175 · 6 operators
0.6%
of known AI assistants advertise signing keys. They are the newest kind of agent and the easiest to check.28 of 38
74%

Sitedex probed 344 of the top retail homepages. A third of them blocked AI agents that their own robots.txt welcomes. We wanted to know what a site in that position can actually check about the agent at the door. Across 2,175 known bots, the answer is almost nothing.

The newest agents are the easiest to check.

The table shows the share of each class that advertises a Web Bot Auth key directory. AI assistants, the newest class, mostly ship keys. Crawlers that have run for years publish nothing to check.

ClassAdvertise keys
AI assistantsBrowser agents and agents that fetch a page while answering someone. Sitedex’s probes imitated these.n = 3874%
AI search crawlersThey index pages for answer engines that cite and link to sites.n = 1127%
AI training crawlersThey read in bulk to build training data.n = 1048%
SEO crawlersThe largest class in the registry. None advertise keys.n = 5450%
D3 bot registry, August 2026 snapshot. Blocking rates: Sitedex, 344 e-commerce homepages, August 1–2, 2026.

An agent can prove its name by signing each request.

The agent signs every request it sends. Its public keys live at a fixed address on its maker’s domain, and that key directory is signed too. Serving it proves the maker controls the keys, which a copied file can’t. This is Web Bot Auth (RFC 9421). A site can check it on the first request, from any address.

GET /product/42 HTTP/2
user-agent: ExampleAgent/1.0
signature-agent: "https://agent.example.com"
signature-input: sig=("@authority" "signature-agent")
  ;created=1754092800;keyid="poqkLGiym…"
signature: sig=:jdq0SqOwHdyHr9+r5jw3iY…:
A signed request. The site fetches the keys once, from agent.example.com/.well-known/http-message-signatures-directory.

Newer standards build on the same proof of domain. ANS gives an agent a name under its maker’s domain. That name leads to certificates and a log of the agent’s history that can’t be quietly edited, so a site learns what the agent is and how it has behaved, as well as who signed the request.

Three quarters of known AI assistants already publish keys. Checking hasn’t caught up. Sites still match visitors against IP allowlists, and an allowlist today covers six companies.

What a check can conclude

What the site seesResult
A valid signature, with the keys served by the domain the agent namesProven
The name and the address both match the ranges its maker publishesProven
Only the name, from a maker that publishes nothing to checkDerived
A known bot’s name, from an address outside its maker’s published rangesSpoofed

Data: the D3 bot registry, August 2026 snapshot of 2,175 known bots merged from Cloudflare Radar’s bot directory and the open crawler-user-agents corpus, with our curation. Verification counts what operators advertise (a Web Bot Auth key directory, or a machine-readable IP-range file), not what validates. Denominators: 79 of 2,175 advertise key directories; 13 (9 agent families across 6 operators) publish IP ranges; none do both; class shares carry their n in the table. Blocking figures: Sitedex’s three-reader study (August 1–2, 2026), 344 leading e-commerce homepages, robots.txt permitting in every case. Analysis and conclusions here are our own.

DNA runs this check on every agent that visits your site.

Sign up free

DNA is free for every site.

It checks signatures against the maker’s keys, and addresses against the ranges the maker publishes. Check your robots.txt