Have you ever had one of your sites get hacked?
bjw: Yes, due to the first OpenSSH remote exploit. With some many servers to manage one of them was forgotten and hacked.Our recovery was a bit complicated, because we didn't trust any of our boxes after that. So, we shut down the hacked box, mounted somewhere else, removed all the data and reinstalled. We reinstalled all of our other boxes too... Lots of work and lesson learned.To secure our systems now we take multiple approaches:-Different operating systems across our network. For example, our web server runs on OpenBSD and our database on Linux (RHEL). We also have Debian and FreeBSD.-Reduce the functionality of each system and who can access it. Plus, never allow SSH from one critical system to another.Added monitoring. We can't be 100% safe, so we started using:-http://modsecurity.org to filter web attacks-http://ossec.net to monitor logs and file changes-SeLinux enabled on all our Linux systems-systrace on all our BSD boxes-http://sucuri.net to monitor our DNS, Whois and site.