[Beowulf]Infrastruture planning for small HPC 40/100 gigabyet eyhernet or Infiniband?

[Gerry Creager]

Er... Fiber's no longer exotic.  I've got Cat6 in the walls of my house 
(although I had to do it as Eugen did by drilling holes and pulling with 
snakes in this house.  In my garage workshop, I've got electrical in 
conduit, Cat6 in conduit and multimode in conduit.  I can add more 
conduit and single mode if I need to later, easily, since the conduit 
resides on the wall surface (it IS a workshop, after all).

We're designing our next/final house.  I intend to have a data race with 
at least 4pr of single mode and multi-mode each, and probably a 4-in 
(10cm) conduit for copper data cables: expansion room.  I can handle 
most anything I can currently imagine with that infrastructure.  I'll 
also stand up wireless for commodity connectivity for visitors, laptops, 
game systems, etc.,

I'm leery of quick-splicing fiber for 10GBE but I'm not uncomfortable 
with that for gigabit or there abouts.  I can order 4-pr 
single/multimode for something around $2/ft these days with 
terminations, in custom lengths.  That's a lot of potential bandwidth. 
Home-run it to the machine room where my home cluster resides, and where 
  the demarc is to the real data connectivity and I should be set.

I've never been afraid to cut holes in walls for connectivity.  I do 
find it easier to do before we've got all the junk in.  In our current 
house, I got to do 3 days of wiring before we moved in, which helped a 
lot.  I got all the cat6 pulled (telco is also on cat6 but on different 
runs from data; no, I'm not doing VoIP at home yet but that expansion is 
simple here).

Nifty niftyompi Mitch wrote:
> On Fri, Jul 25, 2008 at 10:28:39PM -0700, MDG wrote:
>>
> .....
>>    InfiniBand was declared all but dead  not too long ago and it has been
> .....
>>> Fiber is a commodity.  Perhaps you were looking for pricing close
>>> enough to twisted pair copper? In any case, it's not just the cost per
>>> length of cable, the endpoints for fiber are also more expensive.
>> Right now you need GBICs, a splicing cassette, and a splicer
>> (a 10 k$ device). The future looks very bright for polymer fiber,
>> which can be processed with a simple sharp knife (100 MBit/s
>> Ethernet kits + converters are reasonably cheap, 1 GBit/s
>> is being developed -- 10 GBit/s might be rather challenging,
>> unless there's photonic crystal technology).
> 
> Did I read this correctly? 
> Are you cutting holes in the walls of a home.....?
> 
> It seems to me that today's 'exotic' links should be confined
> to the machine room/ data closet.   Consider the current
> maximum length of Infiniband DDR over Cu ....
> 
> After that you can look at end point bandwidth to an office or
> entertainment room.   It is not hard or expensive to pull high quality
> TV coax which can support multiple HDTV links and multiple cat6 for
> trunked data links leaving the machine room.
> 
> Also all wires in the wall MUST be safety rated for such use.
> 
> 
> 

-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843