[Beowulf] advice for newbie

[Vincent Diepeveen]

Look at ebay, there is cheap machines there $150 or less for 8 core  
Xeons L5420,
i'm using those as well. For that $150 you get 8GB ram as well.

Boot them over the network.

That's 16 * $150 = $3k.

Now there might be a difference between my and your wishes.
For me the network is important so i bought pci-e 2.0 motherboards;

In short i bought each component separate which is more expensive  
than $150.

And those rackmounts make huge noise; i'm doing it here with 14 CM  
fans that's not so noisy,
but i assume your institute has a spare room that may make people who  
walk their deaf before
they're 30.

As for you, buy 8 drives SATA from 2 TB and a $30 raid card from ebay  
Those raid cards 2nd hand
are dirt cheap especially the pci-x ones and with 8 drives you won't  
get a bigger bandwidth
anyway and most those online nodes have that.

They're 100 euro a piece here those drives. So that's 800 euro = $1k  
or so. And $30 from
ebay for a total superior raid card. Put them in a ready rackmount  
that allows such drives,
it's a couple of hundreds of dollars on ebay, with the same L5420's  
and motherboards and 8GB
ram. So say a $500 for that rackmount with drives you can plug in.

Put in the drives, build a raid6 partition and your fileserver , the  
17th machine, it's ready to serve you
at around a 700MB/s readspeed.

Now i don't know the latest about genome research; the last Phd  
student i helped out
there, his university used 90s software to do his research.

That really required big crunching for months for each calculation,  
at hundreds of cpu's,
yet new commercial software finished within 15 minutes each  
calculation at a single core.

That 90s software uses MPI if i recall but that'll depend upon what  
sort of software your
guys want to use.

You might want to decide next to buy the cheapest gigabit switch you  
can get, in order to
boot all the nodes over the network using pci-e.

It's possible those motherboards won't boot over infiniband, some might.

Then i'd really advice you buy a cheap 2nd hand switch infiniband,  
maybe DDR, of $300 or so.
Cables $20 a piece times 17 = $140, and a bunch of 2nd hand DDR  
infiniband cards and put
each one in each machine.

So after boot over the gigabit switch, assuming the motherboards  
don't boot over infiniband,
they might boot actually over infiniband in which case you don't need  
the gigabit switch,
then in that case infiniband will take over and can serve you either  
as 10 gigabit network
cards or for the MPI that much software in that area needs.

So all what's left is buy a 17 infiniband cards DDR 2nd hand off  
ebay. Not sure about prices,
maybe a $80 maybe $100. a piece. Let's say it's under $1600

Now you're done with a 16 node cluster with 17 machines from which 1  
is fileserver,
for a part of the budget you had in mind.

Just it's noisy and loud.

Also it's pretty low power compared to alternatives. It'll eat a 180  
watt a node or so under full load. It's 170 watt a node
here under full load (but that's with a much better psu).

As for software to install in case you decide for infiniband, your  
choices are limited as OpenFED doesn't give
much alternatives.

Fedora Core or Scientific Linux are for free and probably your only 2  
options if you want to use free software
that are easy to get things done as you want to.

Then install OpenFED that has the openmpi and other infiniband stuff  
for free.

Probably Debian works as well, provided you use the exact kernel that  
OpenFED recommends.
Any other kernel won't work. So you have to download some older  
debian then, get the exact kernel recommended
and then i guess OpenFED will install as well.

Good Luck,
Vincent

On Aug 20, 2012, at 6:55 AM, Duke Nguyen wrote:

> Hi folks,
>
> First let me start that I am total novice with cluster and/or  
> beowulf. I
> am familiar with unix/linux and have a few years working in a cluster
> (HPC) environment, but I never have a chance to design and admin a  
> cluster.
>
> Now my new institute decides to build a (small) cluster for our next
> research focus area: genome research. The requirements are simple:
> expandable and capable of doing genome research. The budget is low,
> about $15,000, and we have decided:
>
>   * cluster is a box cluster, not rack (well, mainly because our  
> funding
> is low)
>   * cluster OS is scientific linux with OpenMPI
>   * cluster is about 16-node with a master node, expandable is a must
>
> Now next step for us is to decide hardwares and other aspects:
>
>   * any recommendation for a reliable 24-port gigabit switch for the
> cluster? I heard of HP ProCurve 2824 but it is a little bit hard to  
> find
> it in my country
>   * should our boxes be diskless or should they have a hard disk  
> inside?
> I am still not very clear the advantages if the clients has about 80GB
> hard disk internally except that their OS are independent and does not
> depend on the master node, and maybe faster data processing  
> (temporay),
> but 80GB each for genome research is too small
>   * hard drives/data storage: we want to have storage of about 10TB  
> but
> I am not sure how to design this. Should all the hard disk be in the
> master node, or they can be on each of the node, or should it be a  
> NAS?
>   * any recommendation for a mainboard (gigabit network, at least 4  
> RAM
> slots) about $200-$300 good for cluster?
>
> I would love to hear any advice/suggestion from you, especially if you
> had built a similar cluster with similar purpose.
>
> Thank you in advance,
>
> Duke.
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
> Computing
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf