Mixed-Use Bots
Google sees roughly twice as much of the web as any other big AI company. The reason is a quirk in how its crawler works, and it's a bigger deal than it sounds.
For background, Cloudflare put out a report last week about how AI is reshaping the economic model of the web. Cloudflare runs the infrastructure behind a large share of the internet, so it has a bird's-eye view of web traffic (and how much of it is now bots).
Most of the report is focused on the future, but one specific topic stood out to me as relevant today: mixed-use bots.
For years, bots did one job: index your site for search so you'd show up in Google and get visitors. AI changed that. Now a bot showing up at your site could be doing any of three things:
- Search 🔍: indexing your page so people can find it later. This is the one that sends readers to you.
- Training 🧠: pulling your writing in to train an AI model.
- AI answers 💬: grabbing a snippet to answer a live user query.
The difference matters. Search brings people to your site. The other two mostly don't. Your words get used and nobody shows up.
Most AI companies handle this cleanly. They run separate bots for each job and label which is which, so you can keep the search traffic and switch off the training. That's a fair trade.
A mixed-use bot won't do that. It's one crawler doing every job at once. It never tells you why it showed up, and you can't approve one use while blocking another. It's all or nothing. And these bots aren't rare: Cloudflare says more than a third of crawler traffic now works this way.
Google runs the biggest mixed-use bot there is. And it still sends roughly 88% of the web's referral traffic, so blocking its crawler means vanishing from search. Almost no site can do that, so you let it in, and while it's indexing you for search, it's feeding Google's AI at the same time. Both from a single visit you couldn't turn down. That's how one company ends up seeing twice as much of the web as its rivals.
It's a classic bundling move: take the thing people can't say no to (search) and staple it to the thing they'd often decline on its own (training). 📦
The rest of the industry is slowly moving toward bots that say what they're doing. Google, the one bot you can't afford to block, is the one saying the least.