[Beowulf] Virtualisation and high performance interconnects.
patrick at myri.com
Thu Nov 1 07:02:58 PDT 2007
andrew holway wrote:
> I'm trying to find out about the effects of virtualisation on high
> performance interconnects. Effects on latency and bandwidth.
Virtualization has virtually (pun intended) no effect on OS-bypass
(user-level talking directly to the virtualized hardware) operations.
However, it does for all non-OS-bypass operations. The most important of
these is memory registration, because it's not a one-time cost as the
Ohio State papers have always portrayed it. For applications that do not
effectively reuse buffers, or when you cannot hijack the native
malloc/munmap/sbrk symbols, or when you cannot guarantee that the
memory is going to stay around (OS buffer cache for example), the memory
registration (and deregistration !) is going to happen in the critical path.
If the memory registration is ~150us for 1 page, then adding 50us of
hypervisor/IOMMU overhead seems relatively cheap. If the memory
registration is ~2us for 1 page, then it's bad. To give you an idea of
the impact on latency/bandwidth, the communication time (DMA + wire +
DMA) for a 4K page is in the order of 10us at 10Gb/s.
More information about the Beowulf