[Beowulf] MIPS/Watt data
bari at onelabs.com
Wed Jun 23 11:13:32 PDT 2004
We're looking at building some large clusters now using single and dual
core 64bit MIPS SOC's:
Another way to go currently is to use the IBM Blue Logic® SOC:
These two solutions would be much lower Watts and $$/FLOP than x86.
The Via C3 parts are dropping in price but they don't offer much in
cache sizes and I/O throughput. If your code fits into their cache and
doesn't need to be passed around with other nodes much they aren't a bad
Jim Lux wrote:
> At 11:24 AM 6/3/2004 -0500, Brian D. Ropers-Huilman wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> IBM's BlueGene systems are based on similar architectures, 500MHz
>> full system on a chip design.
>> On 2004-06-02 09:57 (-0500), Bari Ari <bari at onelabs.com> wrote:
>> ] Jim Lux wrote:
>> ] > If we compare Dhry mips to Bogomips (who's to say if we're not
>> within an
>> ] > order of magnitude).. A 3.4 GHz P4 turned in 6700 Bogomips. That was
>> ] > probably around 100W (total guess), for 67 Bogomips/watt.
>> ] ARM, Mips and SH all beat the pants off x86 in the MIPS/watt and
>> also in
>> ] the FLOPS/watt department.
>> ] Nobody seems to be interested in clusters built with these
>> ] architectures.
>> - --
>> Brian D. Ropers-Huilman :: Manager :: High Performance Computing
> Not exactly nobody: I'm VERY interested in minimizing joules per
> computation, because every joule and watt is precious in space....
> Lest you think that my application is peculiar, it really isn't, it's
> just an extreme case of the increasing emphasis on total cost of
> ownership. Power consumption is going to be more and more important,
> especially for clusters with more than 4-5 CPUs: Ballpark it.. If your
> node costs $1000/CPU and it burns 100W/CPU. You'll actually need more
> like 150W/CPU (by the time you pump that heat to the big outdoors). Run
> that CPU 24/7 for a year, and you've burned 1.3 MWh, about $250 dollars
> worth, a pretty big fraction of the cost of the node (a REAL big
> fraction if you amortize over 3 years). Cut the power in half to do your
> computation, and you've just effectively bought yourself a bunch more
> There are other capital cost aspects to moving the heat around. Taking a
> quick gander at the Grainger catalog... A ductless split system with
> 23000 BTU/hr capacity runs about $2300 and consumes about 2200W (SEER
> very close to 10). 23kBTU/hr is 6.8 kW. Figuring a 50% overall system
> efficiency, we get pretty close to 3 Watts/dollar. (We'll also assume
> that includes installation costs, etc.) So, to move that 150W out of
> the room costs about $50 in capital costs, as well. Could be twice or
> three times that, or half.
> Before people start beating me up about my pessimistic estimates for
> power and HVAC costs, the real point is that reducing the energy
> consumed to do a unit of computation is a "good thing". There is a
> distinctly non-zero cost to moving the heat away from the CPU to the
> great outdoors that pervades every part of a cluster. You need bigger
> heat sinks or heat pipes and bigger fans, which makes the chassis bigger
> and heavier, which means that the rack has to be stronger, which means
> that the floor has to be stronger, and the room probably has to be bigger.
> For trivial sized clusters, where you can "fit everything under a desk",
> this is less of an issue, but you start to get any size at all, the
> infrastructure costs start to come out of the noise level.
> I think that in the cluster market, this is where vendors are going to
> have to start competing. Clearly, they can't try to make deals on the
> cluster software, because (as evidenced by all the discussion on the
> list of late) the software is basically low cost. A vendor can sell on
> "quality of hardware", but underneath it all, there's only a few mobo
> manufacturers, so it comes down to who gets the best quantity discounts,
> and who has the best sheet metal fabricators, which is a pretty
> competitive business. The basic cost to design, market, and fabricate
> nodes and racks for a cluster is pretty much the same for all vendors.
> So, how, as a vendor, do you differentiate yourself from all the rest?
> (especially if someone wants to make a decision based on quantitative
> - service: repair frequency and speed, documented low MTBF and MTTR,
> etc. - these are tough to quantify in any meaningful way, because
> clusters are still essentially "one-off" unique items, so statistics are
> not particularly meaningful. This comes down to the fuzzy things like
> "vendor reputation".
> - unique value added: Training, preconfigured systems, etc. - again,
> very difficult to quantify, because some bright soul at your prospective
> customer will come up with the "why should we pay X all that money, when
> I can download the software, burn the CDS and get it up and running in
> my spare time".. Granted it's bogus, but it happens. A vendor can't
> prove that the "do it yourself" approach is actually more expensive, and
> unless the customer has built some clusters themselves, they won't know
> any better.
> - lower cost of ownership - something that is readily quantifiable! I
> know what electricity costs, what floor space costs, what sysadmin time
> costs, etc.
> James Lux, P.E.
> Spacecraft Telecommunications Section
> Jet Propulsion Laboratory, Mail Stop 161-213
> 4800 Oak Grove Drive
> Pasadena CA 91109
> tel: (818)354-2075
> fax: (818)393-6875
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf