[Beowulf] Building my own highend cluster
diep at xs4all.nl
Wed Jun 28 14:32:48 PDT 2006
Have a FAQ which describes how to boot nodes diskless?
I'm about to build coming months a 16 node cluster (as soon as i've got
for more nodes) and the only achievement in my life so far in beowulf
area was getting a 2 node cluster to work *with* disks.
That's too expensive however and eating unnecessary power.
14 disks is $$$ but more importantly also eat effectively nearly 30 watt a
disk from the power
(maxtors are like 22 watt and that's *after* the psu lost a lot of
Let's save on power therefore!
Further my experiences with NFS is that it slows down the entire network.
My basic idea is to just buy some wood, and mount 7-14 mainboards
with each a dual woodcrest 1.6ghz dual core and 512MB ram (or less
if i can get that RAM a lot cheaper), or when budget is smaller then
simply A64 with a single chip or mainboards with a single woodcrest chip (in
cases dual core of course).
At it, the 'master node' being a dual opteron dual core (or a 15th node
a bit faster woodcrest chips and quite a bit more RAM) which i plan
to connect to 2 switches which can have 8 nodes each, or if i can afford it
it exists a 16 node switch.
Put the wood & mainboards in in the garage then take a big fan which can use
pipe to blow out air and make some filter to allow air getting into the
this filter catching dust.
The big advantage of putting the cluster in the garage is because a big
shepherd will guard it nearly 20 hours a day there.
(Anyway any environmental norm with respect to radiation i don't need to
care about when constructing that supercomputer,
just walking the dog means in this country you break the law; and in the
meantime my government has
12 meters away from that garage 2 x 450 megawatt (MVA) powerlines,
as they don't need to fix any existing bad situations. In fact they're busy
building a new swimming pool for hundreds of kids underneath
those 2 x 450 megawatt cables right now).
So the only questions right now are:
a) which cables do i need for QM400 cards to connect?
b) which SWITCHES work for QM400 cards?
c) do QM400 cards work node <==> node without switch in between (just like
the QM500s worked here like that)
d) will they work for those woodcrest mainboards?
Of course looking for second hand cables, second hand switches?
diep at xs4all.nl to email to for those who have some left.
I've got 17 of those cards so could on paper move to 16 nodes (keeping 1
card in reserve).
SOFTWARE for the cluster:
My plan is to take a look once more again to openmosix and openssi and plan
to modify it, even if i lose a factor
2 in latency to it after modification, if that would give the possibility
for a shared memory supercomputer then that's
quite awesome and preferred. If losses under factor 2 are not possible to
latency then i'll again have to work with the
Programming in MPI/SHMEM is by the way pretty retarded way to program.
Shared memory programming is just *so superior* over that in terms of ease
for the programmer and it also means
you can just recompile your code to work on a quad opteron or 8 processor
opteron without any need to strip
Patching openssi or openmosix is probably more interesting to do than
continueing an old MPI/SHMEM version.
Migration of shared memory by the way is not needed and not preferred for
the approach i use in Diep,
so on paper both OpenMosix and OpenSSI qualify.
If it can work then that'll be a factor 2.8 ^ 7 = 1349 times more powerful
than deep blue at least, and factor 2 more
powerful than hydra and it's very secure and fool proof, just not saucage
Hydra managed to get on CNN regurarly when playing Adams. Amazingly. We'll
hit by september probably indirectly
national TV when selling one of our products and i hope to reach some more
when this cluster works.
Thanks for any suggestions,
----- Original Message -----
From: "pauln" <pauln at psc.edu>
To: "Vincent Diepeveen" <diep at xs4all.nl>
Cc: "Eray Ozkural" <examachine at gmail.com>; <beowulf at beowulf.org>
Sent: Wednesday, June 28, 2006 8:39 PM
Subject: Re: [Beowulf] Ultimate cluster distro
> This isn't really a distribution-related comment but in light of Vincent's
> points I think it's appropriate. We're running diskless nodes from a
> generic root fs ramdisk which is dynamically configured at boot by a
> cfengine script. Other filesystems (ie /usr) are mounted over nfs. I've
> that this combination of pxelinux and cfengine is extremely powerful for
> managing clusters - especially ones that tend to change frequently. paul
> Vincent Diepeveen wrote:
>> Let me kick off with a few points, most likely many will enhance that
>> with more points
>> a) having a compiled driver into the kernel of the network card in
>> this is by far the hardest part.
>> b) pdsh installed at all machines and naming of machines in a logical
>> c) diskless operation at nodes other than the masternode, using local
>> disk only as 'scratch'
>> d) because A usually goes wrong the capability to easily compile a
>> vanilla kernel inside the distribution
>> D is by far most important
>> ----- Original Message ----- From: "Eray Ozkural" <examachine at gmail.com>
>> To: <beowulf at beowulf.org>
>> Sent: Saturday, June 24, 2006 3:06 PM
>> Subject: [Beowulf] Ultimate cluster distro
>>> I would like to make a small survey here to get
>>> a rough idea of every essential detail in a cluster
>>> distro, because I am thinking of writing some add-on
>>> for our linux distribution to this end.
>>> Eray Ozkural (exa), PhD candidate. Comp. Sci. Dept., Bilkent
>>> University, Ankara
>>> http://www.cs.bilkent.edu.tr/~erayo Malfunct: http://www.malfunct.com
>>> ai-philosophy: http://groups.yahoo.com/group/ai-philosophy
>>> Pardus: www.uludag.org.tr KDE Project: http://www.kde.org
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf