[Beowulf] Building a new cluster - seeking some advice
hahn at mcmaster.ca
Fri Dec 21 21:11:38 PST 2007
> 1. I'd like to go diskless. I've never done this before (the other two
> clusters are...diskful?). I've used Fedora on both of the previous
> clusters. Is this a good choice for diskless? Any advice on where to
> start with diskless or operating system choices?
I prefer diskless installs:
- NFS root: fast, can be RO, no significant server load.
- node-specific files on tmpfs: hardly any - pidfiles mostly.
- local disk for swap, /tmp: disks are cheap and fast, why not?
such an approach is really nicely scalable and very pleasant to
maintain. a diskful cluster, by comparison, is often annoying:
disk failures actually matter, and it's not that hard for nodes
to get out of sync. systemimager does a good job of reimaging nodes,
but it's still not quite as "liberating" as just resetting a node,
knowing it's ephemeral...
> 2. Given my budget (about 20K), I plan on going with GigE on about 24
> nodes. Am I right in thinking that faster network interconnects are
> just too expensive for this budget?
Greg's right: buy the right interconnect, not just the cheapest.
> 3. I'll be spending most of my cluster's time diagonalizing large
> matrices. I plan on using ScaLAPACK eventually; currently I just use
> LAPACK/ATLAS and do individual matrices on each node. The only thing
my experience with scalapack and diagonalization is with monster-sized
sparse matrices, which seem to be fairly latency-sensitive. if your
workload is anything like that, gigabit isn't going to scale well,
at least with a conventional mpi+tcp stack. (I'm looking forward to
the OpenMX stack for this reason.)
> * Intel Core 2 Duo E6850 Conroe 3.0GHz ($280)
> * 8 GB (4 X 2 GB) DDR2 800 (~$200)
did you consider AMD? "large matrices" makes me think of memory balance
(bandwidth per flop), where AMD normally leads Intel.
> The motherboard does NOT have integrated video. Will I need video
> output? Can you even build a node without it?
this is a bios issue: will the board boot without a video card?
I guess you can try configuring it with the card, then remove the card
and see if it still boots. I would make sure you can't get integrated
video - these days, such boards are often cheaper.
> motherboards with adequate support for 8GB memory and 1333 FSB don't
> have video.
I would also consider AMD, which has lots of integrated-video options.
> seems like a waste. From reading around, it seems like there is no
> advantage really to DDR3 memory...is that right? Any advice on the
power savings, probably some headroom in clock, but it's really at
the early-adopter stage, I think.
regards, mark hahn.
More information about the Beowulf