[Beowulf]Infrastruture planning for small HPC 40/100 gigabyet eyhernet or Infiniband?

[MDG]

I agree that the high costs in fiber iw the end-points but when cutting open walls, as I am doing for Cat 6, and 10/100/1000 (or gigabit) is available and while maybe still being developed, the question is this.
 
Just like cell phone technology leap frogged in Asia, they are way ahead on 3rd, 3.5 and the start of 4th generation cell technology, I see things there the USA cannot offer or if does it is too expensive to be worth while yet in Japan, Hong Kong and for that matter mainland China it is part of everyday usage.
 
Now I am assuming that the lack of foresight, just look at our freeways whichwere planned with little usage growth, and were outdated by compkletetion, the same thying is already ahppened with 10megabit wired companies, some manage the 10/100, the smart at least in the last year left a groeth path, at least where tey took my advice, and wired CAT 6 backbones to allow 10/100/1000 without problem but as pointed out the price of te fiber cable itself is fallien to bear commodity prices, the endpoints well I still say Moore's Law will make endpoints fall fast also.  This is also because of the very topic of HPCs.  The distributed processing of an HPC is limited bu the slowest link; Intel did not but high-end PIV Xeons and Quad-Cores for fun, the enternal bus of the computers was not able to keep up so teir 2 and teir 3 cache had to make up the shortfall.
 
I agree fiber end-points, just as fibre-channels drive adapters, are expensive, too expensive.  But so is installing something thst wil be onslete, if it is standarized which I do believe Gigabit has been, the problem is the same as the USA POTS (Plain Old Tele Phoe System)which happens to be copper wire and the fact that compaies do not want to update sunk costs if they do not have to; tjhus the slow adoption of Gigabit and hemce the likelyhood that it will be leapfrogged by 10/40/100 Ethernet or InfiniBand fiber/
 
I do not like to plan a system that has no growth path or is obsolete before itis installed.  It does not mean I have to install the endpoints ommediately,  can pull the Cat 6 for Gigabit  and the back up at the sametime.  the question is which is the more likely winner?
 
InfiniBand was declared all but dead  not too long ago and it has been resurrected.  No one really thoygh anyone wold need quad=core computers, but they are here basically because of developing faster single core was getting harder aand harder, and they are distributed processing which is the bery idea of an HPC. 
 
Now if the enternal bus of modern PCs cannot keep up it becomes fairly obvious that a distributed betwork processing system will need as fast of infrastructure as possible and considering replacing this infrastructur is an expensive process it makes sense to plan in groth paths, at least to me, after all I can wait to use alternate paths till end-point prices fall, the question is which path is the most likely?  yes the typical fiber connection tales special equipment to splice amd connect, etc.  I just am trying to not have to cut the walls open again in a year.
 
So without pointing out repeatly the hgh cost os the end-points does anyone habe any insight or something to contribute on what I see as a critical issue in HPC development.  I think we all know fiber is not as cheap as ethernet 10/100 or Cat 5 (or Cat 5e which some can squeeze 10/100/1000 )gigabit)speed from, and Cat 6 while maybe 203 times per yard/meter as expensive as Cat 5 it is far cheaper then cutting and repulling, expenses, even at a thrird the cable price labor kills youon expenses.
 
Therefore my delimea, where will we be in 1 to 2 years and what sppeds. I know I am alrady looking for dual brodband input at home and pricing business packages.  Further as tele-commuting allows employers to off load physical space to wotk at home there will be another huge demand for speed in bandwidgth, I know I am not happy with mycable modem speed. When you combine that aong with entertainment and telephony (VoIP) we are already hitting the intranet (internal network speeds) and Internet speds are congested so the sale of dedicated bandwidtgh as well as connections ill only increase.  
 
Therefore te best comment was I thnk Greg's on UTC (53).  But specifically has anyone done any study, thought or projections on the needs as well as how to prepare, and the methodology to do so, for the next steps?  I know as a business-person that any major project usually requires analysis of the useful life as well as benefits be it financial, a concern of mine such as NPV analysus for the MBAs out there, to infrastructure and equipment needs over the life or I should say projected useful life for the techs out there.
 
So for the techs what and where do you think we are headed and i know Gigbit, 10/100/1000 is a short-term solution, but how shortterm?  Like I said labor is expensive, end-point debates will rageon as the new replaces the old.  I am just trying to develop something that can do the job for maybe 3 to 5 years.  So if you had to pull Cat 6 and a second system at the same time what would you pull and why?
 
Mike
--- On Fri, 7/25/08, Eugen Leitl <eugen at leitl.org> wrote:

From: Eugen Leitl <eugen at leitl.org>
Subject: Re: [Beowulf]Infrastruture planning for small HPC 40/100 gigabyet eyhernet or Infiniband?
To: beowulf at beowulf.org
Date: Friday, July 25, 2008, 1:00 AM

On Thu, Jul 24, 2008 at 03:55:51PM -0700, Greg Lindahl wrote:

> Fiber is a commodity.  Perhaps you were looking for pricing close
> enough to twisted pair copper? In any case, it's not just the cost per
> length of cable, the endpoints for fiber are also more expensive.

Right now you need GBICs, a splicing cassette, and a splicer 
(a 10 k$ device). The future looks very bright for polymer fiber,
which can be processed with a simple sharp knife (100 MBit/s
Ethernet kits + converters are reasonably cheap, 1 GBit/s 
is being developed -- 10 GBit/s might be rather challenging,
unless there's photonic crystal technology).

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a>
http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20080725/e572e6b2/attachment.html>