I have an m1.small EC2 instance that is mostly just running Apache as a web server for several simple PHP web sites (that use RDS as a database). The server is constantly with a very high load average - around 8, and never below 5. This causes my web sites to be annoyingly slow, much more then I'd expect from a supposedly 2.6GHz CPU with 1.7GB RAM.top and friends shows that other then ~50% steal time (AFAIU, time stolen by the hypervisor for other VMs on the same CPU), the rest is almost entirely IRQ time.mpstat says that this time is spend on a very stable 191.59 interrupts/sec (i.e. that number almost doesn't change over time), and according to '/proc/interrupts' these are mostly spend on xen-percpu-virq timer0 and xen-dyn-event eth0.What are these, and how can I get the EC2 instance to lower the load and spend some more time on my PHP sites?
Why EC2 instance spends all its time in IRQ and what to do about it?
linux;virtualization;interrupt;ec2
After looking at it for a while, this seems like a hardware issue - most likely the physical server hosting the VM is under heavy load from other VMs, and the m1 type machines do not get guaranteed performance, so you get whatever CPU time is left on the host. Worse, if your VM is running on CPU core 0, you also get all the interrupt handling on the machine. See this Xen wiki article under HVM VM's first (and possibly only) VCPU is fully utilised.. Also see this AWS support thread for other people with a similar interrupt problem.The only solution at this point is to get your VM migrated to another physical host. This can sometimes be done very simply by stopping the VMs, waiting a few minutes and starting again - if EC2 has assigned another VM to your old host in the mean time, then you can get a different slot and the problem may be solved. If that doesn't work, the best bet is to create a new image from your VM, and start it on another availability zone. This will cause IPs to be replaced and may require updating security groups and firewall rules. Also make sure you stop your VM before you create the image - to make sure that after the image is created and before the new VM is started, your old VM will not generate new data.