[Beowulf] SC|05: will 10Ge beat IB?

[Jeffrey B. Layton]

Mark Hahn wrote:

>>The Chelsio CX NIC is about $800. You correct in that it
>>uses the new "skinny" IB copper cables and connectors. I
>>    
>>
>
>CX4 is similar, but not identical to IB, afaik.
>  
>

Sorry. I meant CX4. I was told that CX4 is the "new" IB cables,
but I'm not positive since I've seen the cables side-by-side nor
the specs.


>>don't know the bend radius of these new cables, but the old
>>IB cables are very sensitive to bend radius (i.e. make the
>>radius of a bend too small and the cables are toast).
>>    
>>
>
>I don't think this is really a big problem.  optics have pretty
>bad failure modes when bent the right way, too.  heck, even 
>gigabit has a bend spec, though it's not often attended to.
>  
>

I've seen the thicker IB cables break at a very high rate when people
don't watch the bend radius. I've also seen optical cables break but
usually when stepped on or crushed, not when bent.

I've bent the daylights out of cat5 cables and never had one fail yet.
I've had the connectors pull out, but that's about it.

The moral is, pay attention to the bend radius requirements for
thick IB cables as well as the CX4 cables (and the thin IB cables
if they aren't the same as CX4) - it will save time, money, and
heartache.

>>To get about 8-12 nodes, you either need to go with a Foundry,
>>Force10, or Extreme 10 GigE line card (i.e. $$$).
>>    
>>
>
>24pt 10G can be done on a single switch now, which necessarily 
>brings its price down - there were a couple examples of this at 
>SC05.
>  
>

Who did these? I didn't get to the whole floor so it's likely I missed
them.

The only 24-port switches I saw where from Foundry and
Force10 and I didn't get any prices. However, the recent prices
I got from these companies indicate that the per port price for
24 ports is really high.

>>Quadrics was showing a new 96-port 10 GigE switch at SC05.
>>    
>>
>
>Myricom offers something similar.
>  
>

However, their switch is mx based. I never ssaw a 10 GigE switch
from them (but I never got to spend much time with them).

>>Another company is developing a new 10 GigE switch ASIC.
>>Fulcrum Microsystems is developing a new ASIC with great
>>performance (200 ns latency) and 24 ports. The general cost
>>    
>>
>
>I asked about this a bit - the latency sounds great, but if you 
>expect a 10G nic to cost ~10 us, it hardly matters whether the 
>switch is 200 or 600 ns.
>  
>

Unless you are doing fat trees with 3 or more tiers. Also, it matters
when the competition is 2-3 times as large.

It's also important for storage IMHO.

>>I talked to Myricom about their 10G solution. A ballpark
>>price for it is about $1,200 a port, but I don't know their 10G
>>switch prices yet, so it's REALLY a ballpark.
>>    
>>
>
>afaikt, you're not obliged to use myri 10g nics with myri's 
>switch (fulcrum's would work, for instance, or one of the others.)
>though if you do go all myri, you get to optionally run MX 
>at better performance than IB...
>  
>

AFAIK, Myricom is only selling switches that do mx. For 10 GigE
switches you have to find another vendor. However, I didn't get a
full briefing from them so I can definitely be wrong.

Jeff