[Beowulf] using extend-reach IB?

[richard.walsh at comcast.net]

-------------- Original message -------------- 
From: Mark Hahn <hahn at mcmaster.ca> 

> I've got an situation where a 20-25M IB cable would be very handy, 
> and as far as I can tell, such cables exist. What's not clear to me 
> is how they work - I think they all have some form of active components. 
> some appear to be copper; others fiber, but all seem to draw a few Watts. 
I can say a few words about optical active cable (OAC) choices.  The current in production choice is from Intel, their Connects Cable.  This is a VCSEL laser multi mode fiber design with a CX4 connector that runs at DDR speeds out to100m.  A 25m cable is going to run you about $300 (US).  Power consumption at each end is 1.1w.  It is plug compatible with CX4 NICs that have power. Has the small bend radius advantages that you can expect from fiber and other fiber attributes.  There is an excellent study published by ORNL covering their plug-and-play abiliity in the field, performance, and 1000x lower BER (important for full bandwidth utilization).
Another choice, although they are have not fully ramped up production  (Q1 08??) is the Blazar OAC from Luxtera.  This cable is based on a DFB laser and single mode fiber.  It has only three discrete components limited by their very interesting on-chip splitters and wave guides which are used to bend the normally edge-emitted DFB laser light vertically and into the smaller single mode fiber.  The connector is QSFP, not really that common yet.   Power draw  is about the same as Intel Connects VCSEL.  Range is 300m.  Bandwidth is QDR.  No real in-the-field test data that I know of, but the technology has gotten a lot of notice and I think has been selected for special attention at SC07.  Rumor has it that is is already in at least one HPC vendors interconnect technology.  The transceiver-on-a-chip technology is supposed to be a cost reducer, but that has not been proven.  SMF fiber is much harder to align ....
I don't know as much about 10GBASE-T, but it requires cat 6a or 7 cable, is supposed to have a 100m meter range, but the power consumption at both ends is like 5w until they get the transceiver chips down to 65nm.  The signal correction technology is elaborate and may neutralize the "cheap-transceiver" advantage that copper has had over fiber historically, especially as the above fiber alternatives ramp up production and begin to compete with each other.  Of course, per media foot copper has always (?) been more expensive that fiber.  
Hope that helps ...
rbw
PS  There some very good white papers and studies on both the Intel and Luxtera web sites.
-- 

"Making predictions is hard, especially about the future." 

Niels Bohr 

-- 

Richard Walsh 
Thrashing River Consulting-- 
5605 Alameda St. 
Shoreview, MN 55126 

Phone #: 612-382-4620
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20071010/5064db93/attachment.html>