Insights

How does your robots.txt treat AI agents?

We graded the robots.txt files of the 10,000 busiest sites. Enter your domain to see how your own file scores. The check is free, and you don’t install anything.

  • nytimes.comA
  • forbes.comA
  • bbc.comA
  • cloudflare.comB
  • imdb.comC
  • weather.comC
  • reddit.comD
  • shopify.comD
  • github.comD
  • medium.comF
How ten large sites scored on July 21, 2026, on what their robots.txt declares.

Research

What sites declare, what agents prove, and what blocking costs.

  1. October 2026

    Web traffic nearly doubled, and blocking AI crawlers now costs sales

    What blocking costs    482top sites block OpenAI’s search crawler, which keeps them out of ChatGPT search answersof 5,577 readable robots.txt files
  2. August 2026

    The verifiability gap

    What agents can prove    95.8%of known bots publish nothing a site can check, only a name2,083 of 2,175 bots
  3. July 2026

    Lists of bot names miss the agents that matter now

    What sites declare    71%of sites that block Anthropic’s training crawler have no rule for its agents that answer people598 of 840 sites

Scoring

How we score a robots.txt

Three checks add up to 100 points, and the total sets the letter. We score your site the way we scored the 10,000.

Awareness40 points
Does your file say anything about AI search, agents answering people, and training crawlers?
Consistency40 points
If you block a company’s training crawler, do you also have a rule for its agents that answer people?
Decision20 points
Did you write a rule, either way, for at least one agent that answers people?
  • A85–100
  • B70–84
  • C55–69
  • D40–54
  • FBelow 40

Grades read declared robots.txt policy only, not what a site enforces. The top-10,000 baseline is Sitedex’s July 3, 2026 scan (CC BY 4.0); grading is our own.

See which AI agents visit your site, and what they came for.

Sign up free

DNA is free for every site.