Using hyperthreading on 2 Proc Xeon cluster nodes

[Craig Tierney]

I was wondering if anyone had setup a cluster using
Prestonia Xeon Chips with Hyperthreading turned on.
How did it go?

Although I never believed that I would get the 30% speed
up that Intel has claimed, I figured it might give a few
percent by helping get the other system processes out of
the way of the computational processes.

We setup a rack of 16 Dual Xeon machines, P4DP6
Supermicro motherboard with E7500 chipset, Myrinet
interconnect.

We ran 2 processor hpl on each of the nodes.  usually
12 or 13 nodes ran at the speed they were supposed to,
however 3 or 4 nodes ran about half as fast as they 
were supposed to.  When we reran hpl on all the nodes
we would have slow nodes, but they wouldn't be the same
as the first run.

I then ran another code or 2 nodes (4 processors).  Every
10 timesteps the model printed out diagnostic timings.
Sometimes the compute time of 2 of the processes (that
lived on the same node) would go up.

We turned off Hyperthreading in the Bios.  Run times became 
consistent.

Last time I looked in the linux kernel, the kernel
recognized hyperthreading, but the scheduler didn't do
anything to make sure that processes were split by physical
processor, and not just virtual processor.

Has anyone seen this behavior?  Is anyone running a cluster
with hyperthreading turned on?

Thanks,
Craig




-- 
Craig Tierney (ctierney at hpti.com)