GoogleBot is hitting my server hard - and even though I have set the CrawlRate in Webmaster Tools it is still hiking up the load on my server and slowing down Apache for the rest of the normal web traffic.Is it possible to limit / rate-limit connections per second / minute using UFW based on a user agent string?If not how can I do it for GoogleBot's IP ranges?
Can I limit connections per second for certain UserAgents using UFW?
ubuntu;firewall;limit;ufw
You cannot do this with ufw directly, but you need to add the right iptables rules to /etc/ufw/before.rules.I suggest you to learn iptables. As a (not optimized) starting point something like -A ufw-before-input -p tcp --syn -dport 80 -m recent --name LIMIT_BOTS --update --seconds 60 --hitcount 4 --rcheck -j DROP-A ufw-before-input -p tcp -dport 80 -m string --algo bm --string NotWantedUserAgent -m recent --name LIMIT_BOTS --set ACCEPTcould work, where you of course need to replace NotWantedUserAgent with the correct one.This rules should limit the number of new connections per minute from a specific bot - I have not tested them and do not know if they really reduce the workload from a specific bot.