[Beowulf] What services do you run on your cluster nodes?

[Patrick Geoffray]

Perry E. Metzger wrote:
>> You realize that most big HPC systems are using interconnects that
>> don't generate many or any interrupts, right?
> 
> Of course. Usually one even uses interrupt pacing/mitigation even in
> gig ethernet on a modern machine -- otherwise you're not going to get
> reasonable performance. (For 10Gig, you have to do even uglier
> tricks.)

What Greg is trying to say is that high-speed interconnects used in HPC 
do not raises interrupts at all. Data is delivered directly in 
user-space, and the app (or the communication library) busy polls on it, 
no kernel/OS involved. There is one app process per core (usually bound 
to improve locality in a NUMA architecture). When a daemon wakes up, it 
will preempt a core, and the app process just has to wait. If the app is 
tighly coupled, that will delay everybody.

You can say that a daemon waking up every couple of hours is no big 
deal. However, if these events are uniformly distributed on a couple 
thousand nodes, it will happen a couple thousand times more often. You 
can solve this by gang scheduling the daemons across all the nodes, or 
you can turn them off.

However, it is only important for large machines with tightly coupled 
codes. For the majority of the cases, it's just being anal.

Patrick