custom cluster cabinets (was Re: 1U P4 Systems)

[Bari Ari]

Josip Loncaric wrote:

> Bari Ari wrote:
> 
>> I did a quick compute of the P4 thermal design and it would be possible
>> to put 8 P4s into a 1U if you could use the entire surface area of the
>> top of the enclosure as a heat sink [...]
> 
> 
> A neat idea.  However, one would also have to justify its cost.

We don't envision very dense clusters being priced above the cost of off 
the shelf built clusters. The cost of building one enclosure is less 
than eight enclosures.

> 
> How large would the market be? 

Very large if it kept in the same or lower price range as OTS.

 Deep pocket customers only? 

There is no reason why dense clusters will cost more than large clusters 
built using commodity parts. In fact just the opposite will be true with 
nodes built using commodity parts from the component level rather than 
from the board level. We are working on a scalable design now that will 
pack 1TFLOP into a single 19" rack for under $1,000,000 and 64 node 
clusters that will fit under a desk.

Then, the
> price will be high, and all but the most space constrained users will
> flee to cheaper alternatives.  The same reasoning applies to the
> discussion concerning 1U vs. commodity cases.
> 
High performance computing design used to be very costly and time 
consuming. I am still surprised to hear that some CO's still spend 
 >$100K for a new mainboard design and take half a year to complete it. 
Chipsets have become so highly integrated now that mainboard designs can 
be completed in only a few short weeks for only $10-20K. If you're 
building 100 or more systems which is what clustering is all about. Gone 
are the days when custom nodes = high costs.

> Technical computing users used to be kings of the computing jungle, but
> that was decades ago.  A few of them still have deep pockets and the
> ability to buy exactly what they want.  The rest are using mass market
> leverage to buy compute cycles at a discount.  Unless some day mass
> market switches to high density packaging, this feature will continue to
> cost extra.
> 
> I see the Beowulf concept of using commodity components as a way of
> establishing the base price of computing.  I'd be willing to pay more,
> but only if this alleviates some critical constraint.  Then, the price
> of non-commodity extras (Myrinet, high density packaging, etc.) can be
> justified.
> 
With the dropping costs of CPUs and chipsets what we are now seeing is 
that the cost of the Myrinet and SCI parts are several times the cost of 
the CPU and chipset. Maybe someone will make a low cost low latency high 
bandwith ASIC soon to fit the growing market.

Bari