Virtualization Success Is In The Cards

One key to hosting virtual machines on a single enterprise system is to share PCI cards, like a 10-Gbps Ethernet card, while still writing directly to the card rather than having the hypervisor manage all I/O through the card. After all, sending data through the host operating system, meta device driver, hypervisor, virtual switch, and hypervisor driver -- then having the return data work its way through the same I/O stack -- adds latency. And all those context switches eat CPU cycles.

The problem with PCI card sharing is that each virtual machine's operating system believes it has exclusive ownership of all the I/O devices it can reach. A VM will write data to a PCI card and expect data to still be in the card's buffers when it accesses that card again. If another VM accesses that card in the interim, it won't be in the expected state when the first VM accesses it again, and data may be lost.

Recognizing these problems, the PCI-SIG industry consortium has developed a pair of I/O virtualization specifications to describe how I/O devices can be shared between host operating systems. The Single Root (SR-IOV) standard specifies how multiple guests on a single server with a single PCI Express (PCIe) controller or root can share I/O devices. A 10-Gbps network interface card that supports SR-IOV can present multiple virtual NICs to VMs with different virtual LAN locations, quality-of-service policies, MAC addresses, etc.

Multiple Root IOV (MR-IOV) extends the concept to allow multiple independent systems, with separate PCIe roots, to connect to I/O devices through a switch. MR-IOV systems use the same cables and connectors as other external PCIe devices. Both SR-IOV and MR-IOV require explicit support from I/O cards.

Although not every I/O card aimed at servers supports SR-IOV, most 10-Gbps NICs and channel network adapters do, as do some newer Fibre Channel host bus adapters and RAID controllers. MR-IOV hasn't been nearly as widely supported, with just a handful of products available -- and no great demand, either. The trend seems to be for vendors like NextIO and Virtensys to fool SR-IOV cards into doing the things MR-IOV was designed for.

Illustration by Jupiterimages

Return to the story:
Tech Road Map: Keep An Eye On Virtual I/O