[Beowulf] Network considerations for new generation cheap beowulfcluster

[Patrick Geoffray]

Hi Jess,

Jess Cannata wrote:
> By too expensive, I mean much more expensive than Gig-E which is "free" 
> on the NIC side and quite cheap on the switch side.

Everything is very expensive when compared to GigE :-)

> What I should have said is that the NetEffect card is competitive as the 
> number of network connections to a process on a single node increases.

Do you mean TCP connections or iWARP connections ? The number of TCP 
connections only matters to TOEs (TCP Offload Engine) that have to keep 
a state in the NIC for each active connection. For stateless offload 
implementations, the number of TCP connections has no impact.
Similarly, connection scaling problem only apply to iWARP and other 
connection-heavy protocols.

>> no usable latency numbers there.  if you squint, it looks like they're
>> claiming latency of around 7 us, which is _not_ competitive with even 
>> myri 2G (nor recent IB nor myri 10G.)
> The numbers that I saw are not on HPCWire. I didn't realize that when I 
> sent the link. I recommend that people check out the NetEffect cards and 
> similar interconnects (low latency 10 Gb Ethernet with iWARP) to see if 

7-8 us is roughly the UDP or TCP latency on any decent 10GE NICs when 
you set interrupt coalescing to 0. For comparison, MX over Ethernet 
(MXoE) with Myri-10G gets 2.3 us MPI latency with a good 10 GigE switch 
(Fulcrum or latest Fujitsu), today.

iWARP is a bad idea poorly implemented. It makes no sense to have TCP in 
the middle for cluster computing. It requires a TOE and the can of worms 
that comes with it. Furthermore, iWARP has serious Send/Recv semantic 
restrictions that requires explicit per-connection credit-based flow 
control. Just what the doctor prescribed...

> original post. What is/will be the new low-cost network solution? It 
> doesn't seem to be Infiniband or Myri-10G since their price doesn't seem 
> to be dropping much.

There is definitively already a price war in the 10GigE market. There 
are far more players than in the IB business. The volume has not ramped 
up yet and prices are relatively stable. However, low-margin shops such 
as Broadcom are joining the party, so I don't expect the market to stand 
still for very long. In the end, price pressure is good for customers, 
but not that good for vendors.

I wholeheartedly agree with you about 10 Gigabit Ethernet having the 
most potential for lower cost in the medium term.

Patrick