Recent server outage and VirtualBox

This server is a Linux x86_64 virtual machine hosted in a Linux x86_64 system which was running Oracle VirtualBox 4.1.8, which was the most current release the last time the server and all the VMs were rebooted. Another virtual machine also running under VirtualBox 4.1.8 apparently caused a resource exhaustion issue which took down all of the virtual machines on the physical server, and even caused some kernel application-crash tracebacks in the system log on the physical host server.  This caused my web site to be down.

You don't usually see VIRTUAL MACHINES get SCSI, SATA, or IDE disk errors when the host hard drives are fine. In this case the resource-hogging VM apparently caused enough memory issues with the host that the other VMs started having disk, bus, or other issues, like these:

ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
ata3.00: failed command: WRITE FPDMA QUEUED
ata3.00: cmd 61/08:00:58:51:e6/00:00:01:00:00/40 tag 0 ncq 4096 out res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata3.00: status: { DRDY }
ata3: hard resetting link
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: qc timeout (cmd 0xec)
ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata3.00: revalidation failed (errno=-5)
ata3: hard resetting link
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: qc timeout (cmd 0xec)
ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata3.00: revalidation failed (errno=-5)
ata3: limiting SATA link speed to 1.5 Gbps
ata3: hard resetting link
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata3.00: qc timeout (cmd 0xec)
ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata3.00: revalidation failed (errno=-5)
ata3.00: disabled
ata3.00: device reported invalid CHS sector 0
ata3: hard resetting link
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata3: EH complete
sd 2:0:0:0: [sda] Unhandled error code
sd 2:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 01 e6 51 58 00 00 08 00
end_request: I/O error, dev sda, sector 31871320
Buffer I/O error on device sda3, logical block 3670315
lost page write due to I/O error on sda3

I tried to restart the virtual machines (which were all running under VBoxHeadless, not the GUI management tool) using "VBoxHeadless controlvm vm-name poweroff" and I got this error message (it was something like this, anyway):

VBoxHeadless: error: Invalid parameter: controlvm

According to a google search I did, there are no search results for anything like that error message. I guess I'm the only person in the world who's had their VirtualBox installation get into such a bad state that it couldn't even control the VMs to reset/poweroff; hence this blog post. If you get this error yourself you should probably check your VM host right away, because it's probably in a bad state and might need to be rebooted.

I updated VirtualBox to version 4.1.14 because at the moment that is the most current release. I hope it turns out to be able to handle this very heavily loaded resource-intensive virtual machine without causing problems for the other VMs and the host system.