Blog - Home

Blocking AI bots

20/09/26 00:53

I previously set up an AI tarpit. However, the bots mostly got bored of it after about two weeks. At the time, I wasn't sure what to do next, and left things alone, but today I decided to start blocking the bots. I learnt that my previous fail2ban setup wasn't actually functional, and how to fix that.

First, I removed the links to the tarpit from the homepage, replaced the fake sitemap.xml with a real (if sparse) one with no link to it, and updated robots.txt to Disallow the tarpit endpoint. Then the goal was to block anything that still tried to access it - so don't manually type in the URL! (I've redacted it on this post but it may be floating around in the older ones).

Then I created a new fail2ban filter and jail. I'll include my broken config here for demonstration, the correction will be explained further down.

    			
[Definition]
failregex = ^<HOST> .* "GET \/tarpit-endpoint.* \d+$
ignoreregex =
    			
    		

    			
[caddy-tarpit]
enabled  = true
filter   = caddy-tarpit
logpath  = /var/log/caddy/access.json
maxretry = 1
findtime = 1h
bantime  = 1h
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 1w
action   = nftables-multiport
ignoreip = 127.0.0.1/8 ::1 192.168.1.0/24
    			
    		

Once I restarted fail2ban, it quickly started racking up the banned IPs, but I saw no drop in traffic on my dashboard. Can you figure out why? I couldn't, and guessed wrongly that it was because fail2ban was having issues interpreting the timestamps in Caddy's JSON logs. So I took a detour to set up an entirely seperate log just for fail2ban.

    			
log fail2ban {
		format transform "{common_log}"
		output file /var/log/caddy/fail2ban.log {
			mode 0644
			roll_at 00:00
			roll_keep 1
			roll_keep_for 24h
		}
	}
    			
    		

This didn't work at first, turns out I had to enable the transformer module with caddy add-package github.com/caddyserver/transform-encoder. The new log had the nice side effect of logging only requests to the main site and not subdomains, which makes crafting regexes a little easier, but when I updated the jail to point at the new log, it did not fix the problem. Next I turned to investigating the firewall rules themselves.

I found that the problem was that fail2ban was blocking the IPs, but only on port 22.

    			
chain f2b-chain {
	type filter hook input priority filter - 1; policy accept;
	tcp dport 22 ip saddr @addr-set-caddy-tarpit reject with icmp port-unreachable }
    			
    		

I spent a while trying to change this - adding ports = 80,443 to the jail config didn't work, nor did using http/https in place of port numbers. I even tried switching from nft to ufw, but that didn't help either.

In the end, I discovered that /etc/fail2ban/action.d/nftables.conf contains port = ssh, while /etc/fail2ban/action.d/nftables-multiport.conf has been obseleted - so it will override the port config and block 22. The right syntax for the action line in the config was action = nftables-multiport[port="http,https"].

As soon as I restarted fail2ban after this config change, traffic immediately plummeted as IPs were finally actually banned from accessing the site:

I repeated this change in my other fail2ban jails that block vulnerability scanner bots. I had noticed these hadn't been effective previously, but hadn't looked into until now.

For the moment the tarpit is actually still live. Currently it's mostly OpenAI and Amazon bots trying to access it - Anthropic backed off remarkably quickly when I updated robots.txt.

Of course, the bots still can access the 'real' site as much as they like - they only get banned for trying to access the tarpit. Blocking them without false positives is a bit harder, and I'm still thinking about how to do that.