I noticed plenty of times already that when host OS is under heavy i/o load all VMs start to incredibly struggle with i/o up to point where they fail to even boot with systemd giving CPU stuck messages and other timeouts. It works this way both under VMWare Workstation and VirtualBox on Linux host. With any guest. Basically if host is performing for example filesystem check or some checksum computations guest OSes are half-functional due to not working i/o almost at all. It's worth to mention that host processes don't have problem sharing i/o during such load and when second checksumming is run i/o just gets split more or less evenly across 2 processes. Why is that? What makes VM so special? Can it be helped?
Why VMs terribly struggle with i/o when host is under heavy load i/o wise?
virtual machine;performance;vmware;io;storage
Consumer grade controllers/disks have not the capacity to perform multiple heavy I/O operations efficiently ; however as generally as fastest is the technology the process will go better (in theory).Obviously if you running CPU/ I/O intensive operations like software RAID (as the OP said it is doing), it will degrade the performance of the whole setup, both when sharing controllers, and also using up CPU resources. I would advise investing in a hardware RAID controller at least.Server grade hardware will generally deal better with situations of parallel heavy access of several VMs/several users.However, going for server grade hardware is not the whole story, and there are optimisation strategies that are useful both when dealing with consumer or server grade hardware.What can be done in VMs that help a lot into I/O taxing less the hypervisor is paravirtualizing services. Paravirtualization means adding special drivers, that will talk directly with the virtualisation services/kernel for bulk data transfer (aka PVSCSI in VMWare parlance), and as such the actual media storage devices/NICs do not have to be emulated.For all the vmware solutions, either Enterprise or Workstation, you have got for Linux and FreeBSD, the open-vm-tools package.Under Debian, you install it using:apt-get install open-vm-toolsFor Debian Stretch, it does not involve compiling anything anymore. For Jessie, I do recommend installing open-vm-tools from the backports, as the backports install open-vm-tools v10.After installing the open-vm-tools, you have to shut down the VM, and change the disk controller for type ParaVirtual, and the network controller for vmxnet3.Have a look at Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters (1010398)The vmtools also allow the VMs to do memory ballooning and as such, they wont need to use up RAM they are not using.Virtual memory ballooning is a computer memory reclamation technique  used by a hypervisor to allow the physical host system to retrieve  unused memory from certain guest virtual machines (VMs) and share it  with others. Memory ballooning allows the total amount of RAM required  by guest VMs to exceed the amount of physical RAM available on the  host.Support for emulation technologies at the processor level like VT and VT-d also help the process being smoother. Intel Virtualization Technology for Directed I/O (VT-d)Needless to say, optimizations at the OS level that diminish I/O also help; for instance, logging to remote logging systems, and not the local VM. Or aligning partitions. Partition alignment in VMware vSphere 5, a DeepDrive, Part-1Partition alignment in VMware vSphere 5, a DeepDrive, Part-2Beware also of other I/O optimisations, like taking out database storage space out of the /var partition due to the logging daemon flushing log files to maintain log integrity in case of sudden reboots.It also helps following the philosophy of Unix of using the minimum required services. Empirically from my use, smaller VMs will use less I/O in house keeping/paging. Obviously if you use up more memory than you have, you might have problems with I/O (aka trashing).You can also fine-tune the I/O priority of a particular VM in the hypervisor i.e. giving it higher or less priority. I know it can be done in VCenter/VMWare ESX, probably not in VMWare Workstation as it is a level 2 hypervisor, and as such it is the host OS that deals with managing the I/O operations and slice quotas (more on that later on). It also goes without saying that when using level 2 hypervisors that many of the optimisations we talk here should also be applied to the host OS when possible.VMware hypervisor technology also seems to deal better with high-load I/O in multiple VMs than the alternatives.However, if you are concerned about performance, at least in the VMWare realm, at least for production systems, I would advise going for their Type-1/bare metal hypervisor (ESX or ESXi) instead of using VMware workstation.From Hypervisor:Type-1, native or bare-metal hypervisors These hypervisors run  directly on the host's hardware to control the hardware and to manage  guest operating systems. For this reason, they are sometimes called  bare metal hypervisors. The first hypervisors, which IBM developed in  the 1960s, were native hypervisors.[4] These included the test  software SIMMON and the CP/CMS operating system (the predecessor of  IBM's z/VM). Modern equivalents include Xen, Oracle VM Server for  SPARC, Oracle VM Server for x86, the Citrix XenServer, Microsoft  Hyper-V and VMware ESX/ESXi. Type-2 or hosted hypervisors These  hypervisors run on a conventional operating system (OS) just as other  computer programs do. A guest operating system runs as a process on  the host. Type-2 hypervisors abstract guest operating systems from the  host operating system. VMware Workstation, VMware Player, VirtualBox,  Parallels Desktop for Mac and QEMU are examples of type-2 hypervisors.Finally, there is also the option of going for native hypervisors or containers technologies that not add a layer of emulation to mass storage access, like Xen for Linux hosts in PV mode , docker or FreeBSD jails. This alternative also has their advantages and disadvantages, that are out of the scope of this answer.