[Beowulf] Virtualisation and high performance interconnects.

[Patrick Geoffray]

Hi Andrew,

andrew holway wrote:
> I'm trying to find out about the effects of virtualisation on high
> performance interconnects. Effects on latency and bandwidth.

Virtualization has virtually (pun intended) no effect on OS-bypass 
(user-level talking directly to the virtualized hardware) operations.

However, it does for all non-OS-bypass operations. The most important of 
these is memory registration, because it's not a one-time cost as the 
Ohio State papers have always portrayed it. For applications that do not 
effectively reuse buffers, or when you cannot hijack the native 
malloc/munmap/sbrk symbols, or when you cannot guarantee that the 
memory is going to stay around (OS buffer cache for example), the memory 
registration (and deregistration !) is going to happen in the critical path.

If the memory registration is ~150us for 1 page, then adding 50us of 
hypervisor/IOMMU overhead seems relatively cheap. If the memory 
registration is ~2us for 1 page, then it's bad. To give you an idea of 
the impact on latency/bandwidth, the communication time (DMA + wire + 
DMA) for a 4K page is in the order of 10us at 10Gb/s.

Patrick