Publishers block AI training bots, but embrace search

Data: Axios research; Note: "Mixed" includes sites that block/allow only part of their site and/or only select bots; Chart: Erin Davis/Axios

More than 40% of top U.S. publishers block AI training crawlers, compared with just 25% that bar AI search crawlers, according to an Axios analysis.

Why it matters: The data suggests publishers want their content to be discovered by generative AI search interfaces without giving it away to train the models powering them.

How it works: Publishers use robots.txt files to tell crawlers which parts of their website they can access.

Robots.txt files have long been used to manage search engine crawlers, but AI content scraping requires a much more nuanced playbook.

An evaluation of what those files allow today shows which types of data publishers are most focused on restricting in the AI era.

By the numbers: According to an Axios analysis of robots.txt files across 102 top U.S. publisher domains, 31% apply partial restrictions to AI training crawlers, while 39% do so for AI search crawlers.

Search crawlers face far fewer outright blocks than those used to train large language models.

For example, the analysis found that OpenAI's training crawler is more than three times as likely to be completely blocked as its AI search crawler.

Anthropic's Claude training crawler is around 2.5 times more likely to be completely blocked compared with its search counterpart.

Search crawlers, which scrape websites to answer user queries, typically aren't blocked outright. The vast majority of publishers partially restrict OpenAI's search bot (78%) and Claude's search bot (73%).

Between the lines: Publishers are more likely to single out training crawlers from certain AI companies over others.

OpenAI's ChatGPT training bot is explicitly named in 56 publishers' robots.txt policies, compared with 41 for the company's search bot.

A quarter of top publishers don't explicitly address AI training in their robots.txt files, compared with 35% that don't for AI search.

The intrigue: Nearly half (46%) of publishers completely block Anthropic's ClaudeBot, compared with 38% that block OpenAI's GPTBot.

While OpenAI has struck dozens of publisher licensing deals, Anthropic hasn't publicly announced any, per an Axios analysis.

Yes, but: Setting restrictions doesn't guarantee that AI companies will respect them.

Ziff Davis accused OpenAI in a copyright lawsuit of scraping its publications despite restrictions in robots.txt files.

A federal judge dismissed Ziff Davis's claim that OpenAI violated the DMCA's anti-circumvention provision while allowing other copyright-related claims to proceed.

The big picture: Search crawlers allow AI firms to retrieve data to power real-time answers to user queries. This process, called retrieval augmented generation, is what most recent licensing deals are focused on, rather than training.

For publishers with paywalled content, search crawlers are often restricted to content that lives in front of the paywall.

This is why many search crawlers are partially restricted, instead of fully allowed.

What to watch: Publishers are seeking alternatives to manage crawlers at the cloud level.

Cloudflare introduced tools to help website owners manage AI crawlers and explore pay-per-crawl arrangements.

The cloud provider recently introduced a setting that lets publishers block AI training while keeping their content available to be scraped for search results.

Methodology: The analysis studied whether and how publishers grant AI crawlers access to their data using robots.txt.

Axios used data from Similarweb to identify the top 100 U.S. publishers as of August 2026. Two of those publishers, Yahoo and Daily Mail, were observed twice across different domains that appeared on the list: Yahoo and Yahoo News, as well as Daily Mail U.S. and Daily Mail U.K.

Those 102 publisher domains were sorted by how they treat crawlers, as of Oct. 5. Each publisher was classified along four categories: fully blocked, mixed or partially blocked, unrestricted or unknown, meaning the file couldn't be reliably evaluated.

The analysis focused on the same eight crawlers across all publishers, including OpenAI's training, AI search and answers, and retrieval crawlers, Anthropic's training, search and answers, and retrieval crawlers, Google's extended AI crawler and Apple's extended AI crawler.

The study produced 817 observations, which were then analyzed for this story. The vast majority (91%) were readable, meaning the robots.txt files were successfully evaluated.