[Beowulf] Green Cluster?

[Perry E. Metzger]

fkruggel at uci.edu writes:
> Thanks for your suggestions. Let me be more specific.
> I would like to have nodes automatically wake up when
> needed and go to sleep when idle for some time. My
> ganglia logs tell me that there is considerable idle
> time on our cluster.

So probably you don't want to turn off the nodes automatically, you
want to turn them *off*. If your cluster is half idle, why bother
turning it on at all? However, if you want to do this, see below.

> The issue is that I would like to have the cluster adapt
> *automatically* to the load, without interaction of an
> administrator.

> Here is how far I got:
> I can set a node to sleep (suspend-to-ram) using ACPI.
> But for powering on, I have to press the power button.
> No automatic solution.
> Is it possible to wake up a node over lan (without reboot)?

Here is how you can turn them on:
http://en.wikipedia.org/wiki/Wake_on_lan

There is even open source software described on that page.

Again, though, I'd ask if you're really solving the right problem.

If you think you should be "automatically" powering down on average
half the cluster, you could indeed build software to do that -- you
can detect in your job scheduler that many of the machines are
unneeded, ACPI sleep them, and wake them up with wake-on-lan
packets. The software should be straightforward. However, why do you
think you need it? If half the machines are unused, something is
wrong.

If you have a good job scheduler, it should be keeping the nodes
automatically loaded, with a nice queue of waiting jobs. If you're
like most places, there are more jobs that people want to do than
there is CPU. If half of the time there is no waiting job, you're in
an unusual situation. I'd just turn off the half the nodes
permanently or give them to someone who can use them (and that won't
be hard to find).

Generally speaking, if you have a large cluster, and you have enough
work for it, it is going to be running flat out 24x7. If it isn't,
you've bought more hardware than you need.

> How can I detect that a node was idle for some specific time?

If your scheduler program doesn't schedule a node because it has no
work for it, you know it is idle, right? A node could also detect that
its average load was near 0, but that seems like the wrong way to do
it because you want to be scheduling your job queue centrally, so your
job scheduler should both know what is and is not in use, and should
decide on what to shut down.

Again, though, I think you might want to ask if you're doing the right
thing here. If all your machines are not working flat out 100% of the
time, you have hardware depreciating (and rapidly becoming obsolete)
to no purpose, and there are loads of people out there, probably even
on your own campus, who probably are desperate for compute
cycles. (Hell, I could use extra cycles -- I can never afford enough.)

Optimally, a cluster will be working 100% of the time, until one day
when it is obsolete (that is, the cost in space, power, and cooling
is more than replacing it with faster/better hardware), it gets shut
down, replaced, and sold off.

Perry
-- 
Perry E. Metzger		perry at piermont.com