[Beowulf] What is rdma, ofed, verbs, psm etc?

[Brice Goglin]

Le 21/09/2017 17:28, Douglas Eadline a écrit :
>> What about RoCE? Is this something that is commonly used (I would guess
>> no since I have not found much)? Are there other protocols that are
>> worth considering (like "gamma" which doesn't seem to be developed
>> anymore)?
> Gamma has not been around for years. There was open-mx
>
>   http://open-mx.gforge.inria.fr/
>
> But, the project stopped in 2012.
>
> I think the main reason it stopped was that IB is the
> choice of most clusters and most 10G nics provide
> low latency with default tcp/ip (less than 10us in most cases).

No, the reason it stopped is that I didn't have much research to do
with it anymore 
I added support for newer Linux kernels until late 2015 but there's
wasn't any interesting work since 2011.

Another reason is that copy offload became less interesting in hardware
(Intel didn't improve it as much as the rest of the memory subsystem)
and also became difficult to use in recent kernels.


Regarding latency, most NICs had fairly dumb interrupt coalescing a
couple years ago. You wouldn't get less than 10us on pingpong (requires
no coalescing) without killing the stream bandwidth (requires coalescing
unless you can waste CPU cycles). I didn't check recently.

Brice