[Beowulf] SC|05: will 10Ge beat IB?

[Gary Green]

> I wrote an Interconnect survey article about a year ago for
> ClusterWorld magazine. I'm in the midst of updating the
> article and I have a keen interest in the 10 GigE market.
>
> The Chelsio CX NIC is about $800. You correct in that it
> uses the new "skinny" IB copper cables and connectors. I
> don't know the bend radius of these new cables, but the old
> IB cables are very sensitive to bend radius (i.e. make the
> radius of a bend too small and the cables are toast).
>
> Switches are another matter. About a year ago, they were
> very expensive. The NIC/switch combination was about
> $5,000-$6,000 a port from 8 ports up to 128 ports. Now,
> there are some good 8-12 port 10 GigE switches. One is
> from HP and one is from Fujitsu. The per port cost is about
> $1,200. So for small systems (8-12) nodes, the overall cost
> per port is about $2,000 (minus cable costs).
>
> To get about 8-12 nodes, you either need to go with a Foundry,
> Force10, or Extreme 10 GigE line card (i.e. $$$).
>
> Quadrics was showing a new 96-port 10 GigE switch at SC05.
> It uses the Fujitsu ASIC. They use 8 of the 12 ports in the line
> card and then use the other 4 ports to connect the line cards
> together (it's 2:1 over subscribed). They haven't announced
> any prices, but I'm guessing the per port price to be a bit
> more than the cost for the 8-12 port switch.
>
> Another company is developing a new 10 GigE switch ASIC.
> Fulcrum Microsystems is developing a new ASIC with great
> performance (200 ns latency) and 24 ports. The general cost
> for a 24 port ASIC is about $20 a port, so the overall switch
> costs should be lower if the phys costs aren't too high and the
> vendor don't decide to gouge :) It's pretty easy to make a true
> fat tree with up to 288 ports.
>
> So, the overall 10 GigE costs start at about $2,000 per port
> for small systems and move up after that.
>
> I talked to Myricom about their 10G solution. A ballpark
> price for it is about $1,200 a port, but I don't know their 10G
> switch prices yet, so it's REALLY a ballpark.
>
> My general rule of thumb for IB is about $1,000-$1,400 a port
> but the real costs depend upon the configuration and the details.
>
> Hope this helps!
>
> Jeff
>
> P.S. The interconnect survey will be posted on ClusterMonkey.net
> as soon as I can get around to finishing it :)

Jeff points out one of the two issues slowing down adoption of 10GigE in
HPC.  The first bottleneck is the lack of a cheap high port count
non-blocking 10GigE switch.  As Jeff points out above, there is now movement
towards addressing this.

The second issue is the lack of a broadly adopted RDMA implementation.  It
appears that there is movement forward in this arena as well with the move
to adopt the verbs interface from the OpenIB group.

But as someone pointed out in a previous email, just as 10GigE will surely
catch up to where IB is today, by that time, IB will be at the next
generation with DDR, QDR, 12X, etc, etc, etc...

There is also the question as to whether GigE will be able to demonstrate
the ultra low latencies seen in high performance interconnects such as
Quadrics, InfiniBand and Myrinet.  In the end, there will most likely remain
a market for high performance interconnects for high end applications.

-Gary Green