Blog - Home

Site Outage, and tarpit update

03/09/26 04:27

Uh oh. My ISP/router (not sure which) went down from 2:20 to 2:31. That made this site inaccessible. It also meant I couldn't use Matrix, even on mobile data, since I host my homeserver here too. Downsides of self-hosting I guess.

I'm 99% sure this site does not have enough human visitors (or content) that a ten-minute outage is notable. But I do get a lot of robot visitors since I run an AI tarpit, and I figure this is as good an opportunity as any to update on how that's going.

Scraping had been slowly climbing, from around 46,000 requests an hour, to a peak of 53,000 when the outage occured (12-15 requests per second).

Following the outage, activity quite quickly climbed back up to 5 requests per second, and lingered there for a little while, before a rapid jump back up to previous levels.

I'm planning to properly analyse the logs, to find out which bots are most active, where in the directory tree they go, etc. This is quite a big project, as I have gigabytes of logs with millions of unique URI's, so it may take me a while to figure out how to go about doing it.

I can share a few stats by just searching the logs for common user-agents with Loki (just used a line filter) though:

Interestingly, no (or single-digit) hits from ByteSpyder, Perplexity, Petal, Meta, etc. It seems only the big three and MJ12 have discovered my site, and their scrapers are badly-written enough to fall for the tarpit for multiple days. ClaudeBot is particularly aggressive. Fingers crossed, I've saved other sites from ~2 million+ requests (I assume all these scrapers are running full tilt, so every request I take is one less they can use elsewhere).