[Beowulf] Network considerations for new generation cheap beowulfcluster

[Jess Cannata]

I agree that all of the options (Infiniband, Myrinet, and 10 Gb 
Ethernet) are too expensive. I have been looking into the low latency 10 
Gb Ethernet cards from NetEffect, which use the iWARP specifications to 
provide low latency. I haven't done any testing, yet, but the numbers 
that they are releasing show them competitive with Infiniband/Myrinet as 
the number of processes increase. Plus, I expect 10 Gb switches to 
rapidly drop in price. I believe that the only economical solution in 
the short term (3-5 year range) will be Ethernet based since "everyone" 
knows Ethernet. It is only by selling substantial volume that the prices 
drop to inexpensive. I foresee motherboard manufacturers placing 10 Gb 
Ethernet adapters on-board server motherboards soon enough; I see no 
reason why they can't be the low latency varieties.

I hope to start testing some of the NetEffect 10 Gb cards soon and will 
try and post some numbers. Here is a link to some of the numbers that 
NetEffect is publishing:

http://www.hpcwire.com/hpc/716435.html

Jess

Vincent Diepeveen wrote:
> hi,
>
> Thanks for your reaction.
>
> Ethernet is of course too slow in latency.
>
> the cheapest cable i see is 1 meter and $70
>
>
>
> Cheapest card i see is $715
>
> So the node price starts at $765, which is already way way more than 
> the total price of 1 node.
> Now we didn't discuss the switches yet. Switches and routing of a 
> network is important.
>
> The problem of myrinet nowadays is already that it is way too 
> expensive when compared to the node price.
>
> I also tend to remember a few years ago that a myrinet card was like 
> far under $500.
> Now cheapest card of myri i see is $715, and i didn't see the huge 
> price of switches
> yet that will add up to node price.
>
> More interesting than paying a $1000 a node for 10 gigabit MPI, is 
> having some older card say 3 gbit/s,
> which uses MPI and is DMA low latency with a bit older switch for say 
> $300 a node.
>
> Then you've got a good low latency network for a small price, yet 
> still making price of a node more expensive,
> from $450 to $750.
>
> Vincent
>
> ----- Original Message ----- From: "Jess Cannata" <jac67 at georgetown.edu>
> Cc: <beowulf at beowulf.org>
> Sent: Friday, May 18, 2007 10:44 PM
> Subject: Re: [Beowulf] Network considerations for new generation cheap 
> beowulfcluster
>
>
>> While I can't foresee the future, I do think that we are going to a 
>> lot more low latency 10 Gb/s cards that use standard 10 Gb switches 
>> and cables such as Myricom's 10 Gb Myrinet/Ethernet card and 
>> NetEffect's 10 Gb Ethernet card.
>>
>> http://www.myricom.com/Myri-10G/product_list.html
>> http://www.neteffect.com/ne020-features.html
>>
>> Jess
>>
>> Vincent Diepeveen wrote:
>>> hi All of you,
>>>  Now that developments go fast in CPU land, by 22 july or so, intel 
>>> drops price of its quad core to $266 more or less.
>>> Hopefully AMD's quadcore chip releases soon too for a decent price.
>>>  As intels memory subsystem is real weak, not to mention the extra 
>>> price that AMD and intel ask for dual socket/quad socket capable 
>>> chips, the optimal node is a single socket node.
>>>  4 cores is already a lot anyway for 1 highend network card.
>>>  That means in short that you can produce for quite little money, 
>>> far under $500, a node with 4 cores,
>>> or considering the far higher taxrates in Europe, far under 500 euro 
>>> in Europe.
>>>  Basically what a node needs is a mainboard, a bit of RAM, and a cpu 
>>> with cooler. That keeps a node tiny and it's easier coolable. With 
>>> some wood then you can build a great case that holds many nodes. 
>>> Booting of course diskless over the gigabit network. Of course 
>>> interesting to know secondly is whether putting in ECC-reg ram is 
>>> interesting, considering its scandaleous high price always.
>>>  What are opinions here?
>>>  Of course now the question is how to get a reasonable low latency 
>>> highend network with a reasonable bandwidth (latency bigger priority 
>>> than bandwidth of course) and of course being capable of reading in 
>>> memory without writing. Of course the switch/routing prices + cable 
>>> prices need to be included in those price considerations.
>>>  Perhaps some bit older generation card gets sold very cheap now. 
>>> What are the options the coming years there, any manufacturer 
>>> keeping up with the dropped price of a single quad core node?
>>>  Gigabit ethernet is not an option of course, that just works for 
>>> embarrassingly parallel software, it's usually interrupting bigtime 
>>> the cpu and has an ugly one-way pingpong latency, especially when 
>>> there is several threads simultaneously shipping messages.
>>>  What are the options for the network in the future?
>>>  Vincent
>>>  ------------------------------------------------------------------------ 
>>>
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit 
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>