13/09/26 18:39
A while ago I blogged about setting up an AI tarpit. It's been serving a steady 12-15 requests per second for about two weeks now, but I noticed that activity finally dropped off at about 6am this morning.
All told, it's served just shy of 16 million requests in the last 15 days.
I did a quick search for common crawler user agents to see who has the most activity:
| Bot | Count |
|---|---|
| Anthropic | 12,481,794 |
| OpenAI | 2,409,084 |
| Amazon | 504,196 |
| 404,002 | |
| MJ12bot | 26,843 |
| semrushbot | 17,481 |
| Perplexity | 464 |
| ByteSpider | 334 |
| PetalBot | 145 |
| Meta/Facebook | 71 |
| Apple | 168 |
| CCbot | 190 |
| Cohere AI | 49 |
| Total | 15,440,819 |
These numbers are not entirely accurate, as I did not limit results to the tarpit - several were scanning the main site too. Meta/Facebook bots, Apple bots, And Cohere AI bots looked to actually be vulnerability scanners impersonating other bots to keep a lower profile. CCBot results were a mix of impersonators and the real CCbot.
Anthropic makes up the bulk of the requests, with OpenAI far behind in second place, then the other big players like Amazon and Google are barely interested by comparison. I would hazard a guess that the AI companies are using terrible (maybe slop-coded?) crawlers, while Amazon and Google obviously have better tooling and are less likely to get stuck in the tarpit. I'm not sure why the rest of the cohort has so little activity by comparison.
I'm undecided on whether to keep the tarpit running, either as-is or with some bot prevention in place, or just shut it down entirely. On the one hand, it costs me nothing and is useful for monitoring bot activity. On the other, I'm not sure it serves any real purpose beyond that, and it's definitely destroyed my ranking on search engines as I now look like a spam farm. One idea I have is to stop serving it, and block anything that requests it. I'll make a new post when I decide.
I hope to do a much more thorough analysis of the data when I find the time. I've backed up the last two weeks of logs to preserve them from getting rotated out. If you're interested in this data, get in touch! See the homepage for contact details.