I'm trying to understand why I have a huge number of EBS IO request on my Amazon EC2 micro instance.  I launched the instance roughly 6 days ago and has racked up over 4 million IO requests so far.  The instance came pre-loaded with a LAMP stack (Virtualmin, PHP, Apache, MySQL).  I installed one Wordpress site and only loaded it in a browser a few times to do some testing.Can anyone guide me on how to determine what is generating all these IO requests possibly using iotop or some other Linux utility?
Amazon EC2 micro instance large number of IO requests
io;amazon ec2;ec2
I'd add the following 3 tools into the mix as well. Assuming you have them installed, if not you should be able to install them via whatever repository is provided to your ec2 instance.The high load is likely being caused by either disk or network I/O so I'd focus on those 2 areas to start.nethogsNetworking would be my first suspicion, to diagnose that further, I'd use nethogs to see what processes are causing it.ExampleDetermine your network interface, so you can tellnethogs which one to watch.$ ip link show up | awk '/UP/ {print $2}'lo:em1:wlp3s0:virbr0:In my case I'm going to watch my wireless device, wlp3s0.$ sudo nethogs wlp3s0NetHogs version 0.8.0  PID USER     PROGRAM                      DEV        SENT      RECEIVED       2151  saml     /opt/google/chrome/chrome    wlp3s0     2.117       2.715 KB/sec3569  saml     ..4/thunderbird/thunderbird  wlp3s0     0.441       1.496 KB/sec3144  saml     ..aml/.dropbox-dist/dropbox  wlp3s0     0.081       0.061 KB/sec3383  saml     pidgin                       wlp3s0     0.026       0.056 KB/sec4025  saml     ssh                          wlp3s0     0.000       0.000 KB/sec?     root     unknown TCP                             0.000       0.000 KB/sec  TOTAL                                                2.665       4.327 KB/sec Looking at the output we can see that chrome is using the bulk of my bandwidth.iftopYou can see if the traffic is coming from a specific set of sites using iftop.                195kb           391kb           586kb           781kb      977kbgreeneggs.bubba.net        => stackoverflow.com          4.68kb  10.2kb  8.24kb                           <=                            33.5kb  14.7kb  21.4kbgreeneggs.bubba.net        => ord08s12-in-f8.1e100.net      0b   3.90kb  3.99kb                           <=                               0b   3.61kb  3.72kbgreeneggs.bubba.net        => ord08s10-in-f16.1e100.net  5.05kb  4.10kb  5.83kb                           <=                            2.43kb  2.39kb  2.79kbgreeneggs.bubba.net        => stackoverflow.com          1.32kb  3.34kb  4.73kb                           <=                            1.30kb  1.60kb  2.30kbgreeneggs.bubba.net        => cpe-67-253-170-83.rochest     0b   2.19kb   760b                           <=                               0b   2.60kb   862bgreeneggs.bubba.net        => pop1.biz.mail.vip.ne1.yah  5.87kb  1.17kb   301b                           <=                            17.4kb  3.47kb   889bgreeneggs.bubba.net        => 190.93.247.58               480b   2.04kb  2.66kb                           <=                               0b   1.34kb  1.80kbgreeneggs.bubba.net        => ig-in-f95.1e100.net         448b   1.02kb  1.27kb                           <=                             240b    437b    534bgreeneggs.bubba.net        => ord08s12-in-f2.1e100.net    896b    346b    218b                           <=                             480b    221b    124bTX:             cum:    652kB   peak:   85.2kb  rates:   20.6kb  29.3kb  30.1kbRX:                     883kB            161kb           57.9kb  31.4kb  40.6kbTOTAL:                 1.50MB            241kb           78.5kb  60.7kb  70.7kbfatraceYou can use the tool fatrace to see what processes are causing accesses to the HDD.$ sudo fatracepickup(4910): O /var/spool/postfix/maildroppickup(4910): C /var/spool/postfix/maildropsshd(4927): CO /etc/groupsshd(4927): CO /etc/passwdsshd(4927): RCO /var/log/lastlogsshd(4927): CWO /var/log/wtmpsshd(4927): CWO /var/log/lastlogsshd(6808): RO /bin/dashsshd(6808): RO /lib/x86_64-linux-gnu/ld-2.15.sosh(6808): R /lib/x86_64-linux-gnu/ld-2.15.sosh(6808): O /etc/ld.so.cachesh(6808): O /lib/x86_64-linux-gnu/libc-2.15.soWhat else?I'd take a look at this Unix & Linux Q&A that I answered a while ago for more tools to try. It's titled: Determining Specific File Responsible for High I/O.Follow up questions from commentsQ1: Does bandwidth shown by nethogs count against IO requests in AWS? I thought that would fall under 'data transfer' which is a separate category. In iotop the biggest percentage usage was root and a command called 'kswapd0'. mysqld had the biggest disk write usage and httpd had the most disk readI have no idea how this actually is tracked by Amazon. These values are from the perspective of the VM host so they may not correlate even remotely to what Amazon is tracking your VMs usage from their perspective.By the way, this kswapd0 is likely the source of your high IO requests. This is thrashing because, most likely your VM doesn't have enough RAM to satisfy the size/usage of the applications you're running in the VM. So to try and meet the need your system is resorting to making use of swap.You can confirm this a bit more via the free command.Example$ free -ht             total       used       free     shared    buffers     cachedMem:          7.6G       5.5G       2.1G         0B       446M       2.5G-/+ buffers/cache:       2.6G       5.0GSwap:         7.6G        40K       7.6GTotal:         15G       5.5G       9.7GThis shows you how much RAM & swap are in use by your system.Q2: Oh and one follow up question. How does MB or KB of disk read/write in iotop relate to number of IO requests? For example if mysqld wrote 20 M to disk, is there any easy way to know how many IO requests that generated?There isn't really any correlation that I'm aware of with respect to the number of IO read/writes and the aggregate amount of data read/written to disk.Given you're using AWS your actual disk read/writes may very well not even be to a local disk, they could be to storage over the network (SoE - aka. SCSI over Ethernet for example).Your VM would be completely oblivious to this, since the SoE setup would likely be done at the host level and then exposed as disks to any VMs running on the host.Referencesfatrace: report system wide file access eventsfatrace - report system wide file access events