[Beowulf] Distribution / Kernel Memory (STREAM) performance

[Vann H. Walke]

Some of our important applications end up being memory bandwidth bound. 
After upgrading some of our systems to Redhat9 from 7.3, I noticed a
substantial decrease in performance.  STREAM similarly showed that the
Redhat9 system running on identical hardware had worse memory
performance.  

STREAM was compiled statically on the Redhat7.3 system.

Any ideas as to why?  I've tried installing both newer and older kernel
builds on the rh9 system to no avail.  Oddly the rh7.3 system is running
from the same base kernel (possibly redhat patch changes which I haven't
investigated).  

The STREAM results are included below.   Note the Add and Triad numbers
for system #2 (redhat7.3) which are about 10% better than the other
systems.  All systems have identical hardware and bios configs.

Vann 


(All kernels are Redhat supplied except where noted - kernel.org)

======================================================================
System #1 

(redhat 9, 2.4.20-30.9smp)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1417.6783       0.0227       0.0226       0.0230
Scale:       1429.3428       0.0225       0.0224       0.0225
Add:         1607.7183       0.0300       0.0299       0.0304
Triad:       1609.2309       0.0303       0.0298       0.0325

(redhat 9, 2.4.20-8smp)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1331.1701       0.0242       0.0240       0.0244
Scale:       1316.0622       0.0244       0.0243       0.0249
Add:         1632.5440       0.0296       0.0294       0.0298
Triad:       1636.9419       0.0297       0.0293       0.0313

(redhat 9, 2.4.20-8 uniprocessor)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1341.9969       0.0240       0.0238       0.0242
Scale:       1327.3646       0.0242       0.0241       0.0246
Add:         1638.9545       0.0296       0.0293       0.0306
Triad:       1645.9833       0.0293       0.0292       0.0295

======================================================================
System #2 (redhat 7.3, 2.4.20-24.7smp)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1476.5567       0.0218       0.0217       0.0223
Scale:       1491.0847       0.0215       0.0215       0.0217
Add:         1805.8643       0.0268       0.0266       0.0274
Triad:       1802.0035       0.0267       0.0266       0.0272

======================================================================
System #3 (redhat 9, 2.4.20-31.9smp)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1474.7299       0.0244       0.0217       0.0408
Scale:       1482.5777       0.0228       0.0216       0.0307
Add:         1666.3822       0.0293       0.0288       0.0325
Triad:       1674.2198       0.0287       0.0287       0.0288

======================================================================
System #4 (redhat 9, 2.4.24 kernel.org)

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        1331.6683       0.0241       0.0240       0.0243
Scale:       1328.0255       0.0242       0.0241       0.0244
Add:         1682.2652       0.0287       0.0285       0.0290
Triad:       1691.0367       0.0285       0.0284       0.0287