[Beowulf] SC|05: will 10Ge beat IB?

[Patrick Geoffray]

Hi Mark,

Mark Hahn wrote:
> indeed, it's CX4 that I was seeing all over the place.  yes, it's a 
> pretty heavy/thick cable, and not cheap, and limited to ~15M.
> but for clusters, 15M is pretty workable.

The "thin" CX4 cable (a little bit thicker than CAT5 and more rigid") is 
limited to 3M. The "garden-hose" CX4 cable can go up to 10M, after that 
it really depend on the quality of the cable itself (max on paper is 
15M). This fat cable is about twice as thick as the "thin" cable, and 
it's much much more rigid. However, they are usually less expensive than 
the thin cables.

How many nodes can you practically plug in a 3M radius around a switch ? 
In a 10M radius ? That's pretty much the only reason why the Torus 
topology (and all of its associated problems such as high latency and 
link sharing) is back: how to deal with short rigid copper cables.

> is there really a promise of running 10G on something like cat5?

Yes, but I don't think they are there yet. We have evaluated several 
implementations, and they had either excessive power requirement or 
limited cable length.

> my understanding is that an IB cable has some active equalization
> built into the connectors, and that this is the only difference between
> it and eth-style CX4.  I also understand that putting the equalization

 From what I remember, the constraint on the twisting rate are different 
also, because of the higher signal rate of 10G. In short, IB cables and 
10Gbase-CX cables are different, 10Gbase-CX could be used for IB (if 
equalization on the NIC) but not the reverse.

Patrick
-- 
Patrick Geoffray
Myricom, Inc.
http://www.myri.com