[Beowulf] fast interconnects

[Jim Lux]

At 10:58 PM 5/25/2006, Patrick Geoffray wrote:
>Hi Gilad,
>
>Gilad Shainer wrote:
>>>a) There's likely to be 10Gbps ethernet over ordinary cat 5e/6 cabling
>>soon. (Solarflare is one company working on it)
>>It is not a surprise, as you can run InfiniBand on cat 6 cables today.
>>There are several solution in the market that make it happened.
>
>You can run anything on a shoe string between two yogurt cups today, but 
>it does not mean it's a good idea :-)
>
>All Cat6 solutions have the same problems:
>
>* overhead: KeyEye's transceivers for 10GigE (KX1001) adds 150ns. That's
>0.3us for a point-to-point link if you assume it's per chip, twice that
>with a switch in between. The transceiver for IB-SDR (KX1003) adds 200ns, 
>that's 0.4 us per link, 0.8 us with one hop.

I think that for any very high rate interconnect, latency starts to be a 
significant factor (unless your messages are huge).  10G-T signals at 
(IIRC) 800 Mbaud with a lot of bits/symbol (3-4, I think).  However, the 
real latency constraint is going to be the block size for the LDPC coding 
which is 2048 bits.  At 10 Gbps end to end throughput, those 2048bits will 
take 204.8 nanoseconds, no matter how you deal with it. I imagine that the 
decoder has some non-zero processing time.

Some of the competitive interconnects may do less coding (e.g. the cx4), in 
which case the latency through the link could be shorter.



>* power: between 3.5 and 3.7 watts per chip. That's more than the rest
>of the NIC consumption. On a 256-port switch, that's almost an extra 1KW 
>to cool.

To a certain extent, speed costs power, because you need to get a certain 
SNR to support the BER at the receiving end. 10x speed takes (to a first 
order) 10x power (or, more accurately, the energy per bit remains roughly 
constant, because the Eb/No determines the BER)

Figuring around 3-4W per port for 10G-T, what are the competitors (that can 
support 10 Gbps) coming in at?

The Fujitsu CX4 transceiver is in the 2.1 Watt range, so we're not talking 
order of magnitude differences.  I wasn't able to tell in a quick glance 
whether the transceiver provides any of the 8B/10B coding used, I assume it 
does.

I suppose the real question is whether you even want to use Ethernet (in 
any form) at all for these fast interconnects.



>* length: 25-30 meters, not much more than CX4.

I think that the intent of the 802.3 standard for 10G-T is to support 
55-100m on Cat 6, and no less than 100m on "Cat 7"

>Granted, it would
>certainly be a bit thinner than a 20 meters "firehose" CX4 cable, but it
>does not help on the length side.

And the cable costs would be much lower with 10G-T. twisted pairs, for a 
given performance, are going to be cheaper than twinax.


>The only advantages I see are price of the cable (to be confirmed) and 
>look-like-cat5 warm feelings. Both are very important, but not for 
>technical reasons.
>
>Patrick

Jim