[Beowulf] Building a new cluster - seeking some advice

[Jeffrey B. Layton]

Robert G. Brown wrote:
> On Sat, 22 Dec 2007, Jeffrey B. Layton wrote:
>> Robert G. Brown wrote:
>>> On Sat, 22 Dec 2007, Jeffrey B. Layton wrote:
>>>> very small swap file on the thumb drive if you think you need it). 
>>>> There is work
>>>> afoot in the Linux kernel to allow real swapping over the network. 
>>>> It's not quite
>>>> there yet (the last I looked), but it does look like Linus will 
>>>> allow it once it's
>>>> reaches some level of maturity.
>>>
>>> I looked at this a LONG time ago (back when I was running diskless 
>>> nodes
>>> out of sheer necessity because the nodes we got as part of a giveaway
>>> program had an unsupported SCSI controller).  There was a "move afoot"
>>> then, too, but this was maybe 2000.  So don't hold your breath.
>>
>> http://lwn.net/Articles/262379/
>>
>> http://kerneltrap.org/Linux/Swap_Over_NFS
>>
>> http://kerneltrap.org/Linux/Memory_Management_Improvements
>
> Yeah, all that stuff.  Memory management, locking, page sizes, slow, and
> if you pushed it it could lock up your system, but maybe adequate to
> keep your system from locking up if you run barely over, rarely.

I agree. I personally like the idea of stateless nodes (with or without
disks) or a large number of reasons. I know the memory footprint
of the apps I tend to run. But there are people who don't (I'm probably
in the minority actually) and there are times when you get the wrong
number of nodes in the job and it starts swapping. I can also think of
an application I've run that needed more memory on start up and shut
down than while running because the rank 0 did all of the IO for the
rest of the nodes.

The idea behind swapping over NFS is that in these cases when the
apps do swap, they don't die, they just run like molasses in winter.
Hopefully the user or the admin notice this and take action (At one
time I wrote a swap detector that would find apps that are swapping.
This could be added to something like Ganglia, Wulfware, or other
monitoring tool to detect swapping. But to be honest, I'm not sure what
I did with it - plus it may have been expensive in terms of CPU time).
You could just as easily swap to a hard drive, or flash or whatever,
but personally, I like the idea of not having any drives in my compute
nodes if I can help it.

Also, there are users in classified environments who would love to
get rid of as many disks as they can because of administrative as well
as security issues.
> The diffentiation then is management.  I'm not convinced that it will be
> easier to install and manage a cluster with (say) 1 to 4 GB flash drives
> used as boot compared to using e.g. warewulf to manage boot images.  Or
> that it will be faster.  Or (really) cheaper -- $40 is still $40 more
> than a diskless system, and $40 that would buy it more real memory that
> is likely to ultimately be more valuable in terms of improved
> performance and stability.

Good point. I agree :) Now, about teaching you Fortran90....

Jeff