[Beowulf] precise synchronization of system clocks

[Lawrence Stewart]

On Sep 30, 2008, at 11:37 AM, Robert G. Brown wrote:

>
>
> However, I would think it would require a lot of work to get the
> kernel(s) to respect a usec-synchronized clock, assuming that one  
> could
> constrain the hardware so that it didn't generate too much random  
> (e.g.
> interrupt) noise on its own.
>
>   rgb
>
> -- 

I don 't think this is all that hard - once we got the cycle counters  
synchronized
here at SiCortex, the rest of Linux behaves fairly well.

The first trick is to get all the timer interrupts to happen at the  
same time.  On
our MIPS cores, this is easy by using an interrupt generated by a  
comparator
against the cycle counter (architectural feature).  I would hope there  
is something
similar in x86.

Once the timer interrupts are synchronized, pretty much all background  
tasks become
syncronized.  This even works for many device drivers that use the  
device interrupt to
schedule a tasklet or top half.

The next step is gang scheduling of kernel threads, so that all the,  
say, kswapd's system wide
chose the same timer tick to wake up.

The next step is gang scheduling of user threads, so that all the,  
say, cron daemons
wake up on the same timer tick.

Our experience is that just aligning the timer interrupts helps a lot.

-Larry