[Beowulf] First 96-Node Transmeta Desktop Cluster Ships

[Vincent Diepeveen]

At 11:51 AM 5/4/2005 -0700, Jim Lux wrote:
>At 10:08 AM 5/4/2005, Vincent Diepeveen wrote:
>>Yes it is very interesting.
>>
>>However their sales price of $100k they initially quoted i found a bit
>>expensive.
>
>I suspect that they picked the price point according to typical "purchasing 
>authority" thresholds.

A wrong way to price things IMHO. In current economic situation no one will
want to waste money for his company. $100k for such a box, even though it's
an interesting box, is way too much for the performance it gives.

>>I don't want to be rude, but these processors are quite slow if you just
>>look to 1 processor and they go back in performance after a while when they
>>go hot.
>>
>>So a fully loaded system will have a far smaller performance in the long
>>run than it has in the short run.
>
>But then, this isn't a device designed to be churning 24/7 serving multiple 
>users.  It's for someone like me to shove under my desk and use in a very 
>"bursty" fashion.  Much like the processor utilization on my desktop 
>computer now, where it runs around 5% most of the time, and bursts to 90% 
>when I fire up Matlab and start crunching.  But then, I go to lunch, and it 
>drops back down.

Give 1 example of a user that has $100k to spend on a machine, without
actually using it.

Rich Oil-tycoon from texas perhaps?
If so, mind giving me his adress?

oh by the way i'm just building a 3d dual k7 here. 1 dual i try 24/24 and
has a load average of 2.00 2.00 2.00

the 3d one i'm building. This dual is under the heavy load of emails.

>
>
>>Of course that has its charm too, a system that's fast if you just use it
>>real short at a "critical" moment.
>>
>>So the quoted performance is still impressive then.
>>
>>The network is not so impressive what each node is connected to another
>>node. That is not relevant for embarrassingly parallel software though.
>>
>>Relevant is the price per gflop IMHO.
>
>Look at TOTAL Cost per USEFUL gflop.

Oh well, for my chess program it's a wasted $100k, so let's not get in that
discussion.

The network is way too weak. one way ping pong latencies between the
processors i didn't see published, i suspect around 200 microseconds or so?

Because for all the shared memory software it's not a useful machine, just
for embarrassingly parallel software, in general that means that it can run
toy programs.

A toy program i'm working on now is something that's gonna win (in my
dreams that is) : 
  http://www.eff.org/awards/coop.html

He wait a minute!

In order to win $100k i'm not first going to spend $100000 !

>
>>For a NEW product that tries to fight itself into the market, you must
>>simply be factors cheaper than the competition. $100k i find a tad
expensive.
>
>Or, find a new and different market that has different requirements, not 

Perhaps I know 1 market, that's marxistic/communistic/green political
movements, buying the machine, because they are a big fan of a green nature.

So a box that doesn't eat too much power would be accepted there.

Usually such organisations aren't known for buying expensive equipment though.

I remember a campaign: "save the animals" (lekker dier), which my sister
started the marketing campaign for. It was an effective campaign.

Government organisations will not soon buy these boxes, apart from that i
do not know a single government body that would get a budget for a $100k
computer, does the processor have ECC in the L1 and L2 caches?

I agree, if that is the case, that it is a ridicioulous question, 
but they are requiring it for computations.

>currently well addressed by the existing marketplace.  This product will 
>NEVER compete against a rack of 4U chassis with a team of gradstudents 
>installing and maintaining it for (virtually) free.  Nor will it compete 
>against any "home built" system, where labor is free.  Nor will it compete 
>against the 1024 processor dreadnought class clusters.

It's in the same price league like a home build dreadnought.

>
>They'll not sell it on dollars/GFLOP.  They'll sell it on the basis of 
>increased productivity of expensive people.  The structural analyst with a 
>fully burdened cost of $250K/yr who becomes 15% more productive, for 
>instance, will pay for it in the usual 3 year amortization 
>interval.  Justifying that cost/benefit will be the real challenge, but 
>it's no different (conceptually) than justifying a $5K wordprocessor for a 
>$15K/yr typist. (1985 GS-3 Typist II.. probably after you add burden, 
>around $30K/yr cost)

Ah, again you're showing how commercial government officials are.

Believe me, no objective economic measurement will cause a company to buy
this type of machines, as there are cheaper alternatives eating less power,
producing more gflops when you need it.

In fact i was spammed previous week by someone on this list, marketing
pci-cards which they want to sell probably for a price i find too expensive
for a pci-card too. But it is multiplying giant matrice quickly.

Please note that nowadays your salaries for programmers, world wide seen, 
are not true at all.

First of all majority of software does NOT get made in USA.

Secondly in this country, i would really be HAPPY with a contract job to
parallellize software (or whatever algorithmic software) around $30k.

Because of the low dollar it still looks like a lot (1.28 dollar = 1 euro)
30k euro is about what a programmer earns here.
very good project leaders of programming teams are peeking at 42k euro a
year. Usually 10% of that additional to bonus.

Indian and Indonesion programmers, which slowly but increasingly are taking
over majority of software programming, they are working 60 hours a week to
make $12k a year and feel real rich then (and they ARE when compared to
others).

Where in europe and USA additional costs are big usually, in case someone
is getting a salary, they are not for contract jobs and additional
conditions for personnel in the far east you hardly need.

With a k7 they already are REAL happy.

Now you want them to buy a $100k machine that cannot be under 24/24 load?

One of the most important jobs of engineers IMHO is to make things cheaper,
not more expensive.

To give examples, i just bought a second hand dual k7  mainboard + 2x
MP2800 for 325 euro. That was a few weeks ago. 

That single dual k7 delivers more effective work for 99% of all
applications i run than an Orion will.

The only type of software i can imagine to be profitting from Orion is
24/24 embarrassingly parallel software.

>Here's a quote from a 1989 business magazine:"
>When all is said and done, however, forget about features; forget about 
>price. Instead, choose the program that you can get help with at midnight 
>on a Sunday when you have a proposal due at eight o'clock on Monday morning.
>"
>Another report from that era mentions that factories capitalize their 
>workers (in 1981) at $25K/worker (that is, they invested $25K in plant and 
>machinery for that worker), while clerical staff was capitalized at $2.9K.

You left businessworld in 1989 and started a job at government?

here is a wake up call : we are just recovering from an economic recession
here and knowing how completely dried out the future is from the German
economy, also considering tomorrow we celebrate here in Netherlands how
Germany surrendered all forces in the low countries at 5 may 1945, which
won't help german morale either, it is very simplistic to predict hard
years to come. Germany is the central European production country. In UK
and France the picture isn't much different from that. Even a referendum
for a formal constitution of an united Europe will most likely get blocked
by the French, causing even more instability here.

In that political and financial unsure and instable climate, no company
will just throw away $100k for a computer that can work for a few hours.

In general this is true for highend too.

The real expensive deals are over. 

I know a communication giant which provides infrastructure for millions of
users in South America, who has standardized away all expensive Sun boxes,
and replaced them with quad xeons a few years ago with linux and custom
made software. 

REAL cheap. If 1 quad goes down (which hardly happens), 
another one instantly takes over.

I do not know what is the next hardware that will be bought to replace
those 24/24 boxes, and i sure won't find out until it is a run race, yet no
highend will earn much coins there.

Before these boxes were there, real expensive Sun hardware was used.

The savings are tens of millions. Just 1 small department delivering
connections for a few million users.

Such optimizations happen daily nowadays. 

Highend can only keep highend and keep alive if they can deliver huge
performance for a small price. The exceptions to that rule, which in the
80s were not exceptions but the rule, they are slowly dissappearing.

If economy keeps having problems recovering, in the end also government
organisations can no longer waste money and will need to size down. Their
commission payments probably being the last to dissappear. Budget for a new
machine is really the FIRST thing that gets sized down.

>>So to speak you can buy 10 quad dual core opterons 1.8Ghz for that price.
>>
>>that's 80 processors x 3.6 gflop = 300 gflop too.
>>
>>So effectively 10 quad opterons are faster in floating point than this
>>orion machine.
>>
>>It's true however that the 10 quads eat a bigger power bill.
>>
>>But if you can afford paying $100k that won't be a problem either.
>
>
>Au contraire...  The problem with power isn't the electricity cost.  It's 
>the removal of the heat and other infrastructure costs.  Those 10 quad dual 

Houston, we have a heat problem outside!

(loud laughter), it's nearly summer here now. It's 10C outside. Real nice
temperature. If i open the window, all heat here is directly gone from 3
dual machines, each having a 460 - 550 watt power supply. In the winter i
never open the window and turn the additional central heating a bit lower.

>core opterons will probably fill a rack and have a mighty roar of cooling 
>fans, and I won't be able to plug it in in my office built back in the 
>60's.  (assuming I have ear plugs)

Basically i do not know any system >= 4 cpu's that's real quiet.
Well you're deaf anyway, this orion system produces 50-55 decibel,
that's ear deafening.

The loudest fans i have had here were delta fans, still have them lying in
the dust in case in future i have another overclocking adventure. 

They produce a huge engine sound, that's 46 decibel. I had that machine
equipped with them in the attic, but could hear it also downstairs.

A cheap dollar per gflop solution, producing < 28 decibel sure would be nice.

>Lest you say, well, just use it remotely, and park it in the machine room; 
>I'd just point out that the same argument was made when PCs were 
>introduced.  Why use a PC?  We'll give you a perfectly good VT-100 or ADM3, 
>and we'll take care of the machine down the hall, and then, we can all 
>share the massively bigger processor and achieve economies of scale.  We've 

If you have some giant object of 2x2x2 meters that can produce 300 gflop
for me for a small price, like $4000, i know many buyers.

At least my 80 year old neighbour for the first time in her life will
understand that it was a cheap buy :)

Perhaps it will even inspire her to write a new book.

Those giant machines from the 80s didn't go away because they were too big.
Most importantly they left the building because they were too expensive for
the performance they delivered.

Because chips are so much easier to print than big cases with fans and
mainboards, it's not impossible that in the far future a small PDA a look
like console will be replacing our pc's. 

Of course the famous Bill first must get to work voice communication a tad,
as typing and reading email at a pda is not so easy :)

>even set up a very slick batch job utility and your printouts will be 
>delivered to a box down the hall within an hour.  (There is old serial 
>cabling protruding from the raceways behind my desk dating from that very 
>era, a time of Gandalf "short haul" modems and current loop interfaces).
>
>>Nevertheless if we consider the fact that this company just starts to make
>>clustered systems and already so quickly puts such an impressive product in
>>the market they sure have my compliments.
>>
>>IMHO they are the right type of company to make a clustered cell type
>>processor system (if ibm wants to deliver them cpu's). Their system needs a
>>fast processor that uses little power. The philosophy i really like.
>>
>>Please note it would perform horrible for my own software, as it is latency
>>sensitive software, so anything i write down here is meant for software
>>other than my searching software.
>
>Perhaps.... they DO have the advantage of a very wide/fast bus internally 

Perhaps read a few blueprints of the machine. It's all processors connected
with cheapo nic's to each other and after a while it delivers just hardly
more than 115 gflop. 

The new sony playstation for sure will be faster than that.

For $500 :)

>among the processors, with short runs. Since they don't have to go "outside 
>the box", they can control the interfaces: all those line 
>drivers/receivers, and just connector costs add up.  Decent shielded 
>connectors are several bucks apiece, and when you have to buy 100 of them, 
>thats a significant fraction of the purchase price.  Not to mention power 
>distribution, power supplies, and all those things.
>
>
>
>
>
>James Lux, P.E.
>Spacecraft Radio Frequency Subsystems Group
>Flight Communications Systems Section
>Jet Propulsion Laboratory, Mail Stop 161-213
>4800 Oak Grove Drive
>Pasadena CA 91109
>tel: (818)354-2075
>fax: (818)393-6875
>
>
>
>