[Beowulf] building Infiniband 4x cluster questions

[Vincent Diepeveen]

On Nov 7, 2011, at 6:16 PM, Gilad Shainer wrote:

>> They use the term "message oriented" with the description that the IB
>> hardware takes care of segmentation and so forth, so that the  
>> application
>> just says "send this" or "receive this" and the gory details are
>> concealed.   Then he distinguishes that from a TCP/IP stack, etc.,  
>> where
>> the software does a lot of this, with the implication that the  
>> user has to be
>> involved in that.
>>
>> But it seems to me that the same processes are going on.. You have  
>> a big
>> message, it needs to be broken up, etc.
>> And for *most users* all that is hidden underneath the hood of,  
>> say, MPI.
>> (obviously, if you are a message passing software writer, the  
>> distinction is
>> important).
>
> You can also post large message to the IB interface (up to 2GB I  
> believe) and the IB transport will break it to the network MTU.
>
> Gilad
>

Please note that searching only requires massive amounts of short  
data requests, say 128 bytes and massive
stores of 32 bytes. So latency of the network cards and how fast the  
cards can switch from proces to proces,
those latencies play a far more important role than all those single  
core latencies that everyone always posts.

Some cards when switching from helping 1 proces to another can have a  
penalty of dozens of microseconds;
you never hear from all those hidden penalties as the few online  
tests done by all those academics are always
  single core tests with the rest of the cluster idle.

Interesting is to hear experiences there, but reality is that you  
hardly ever hear that. You have to gamble what to
buy.

What i do know is that the MPI programming model is not very  
attractive; you have to catch all those messages
shipped somewhere, check for overflow and so on.

Yet i'm on a budget here, so price dominates everything.


> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
> Computing
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf