[Beowulf] mysterious slow SATA on one machine

[David Mathog]

I'm hoping somebody has seen this before and can suggest what might be
going on.  

One machine (Arima HDAMA-I board, dual Opteron 280, 4GB RAM,
Sil 3114 Sata controller, Sil 5.4.03 firmware) has mysteriously slow
SATA IO.  This is the case for two different disks (WD10EARS and
ST340014AS), two different disk schedulers, and two different OS's
(Mandriva 2010.0 and PLD 2.97 rescue linux.)  Using a different brand of
cable, and plugging into a different SATA port didn't help either. 
However, move those disks to another machine (Asus A8N5X, Nvidia CK804
SATA controller, single core, 1 GB RAM, Knoppix) and they are both much
faster.  Raw results from various experiments here:

  http://saf.bio.caltech.edu/pub/pickup/bonnie++.rtf
  http://saf.bio.caltech.edu/pub/pickup/sustained_write.rtf

For the sustained write test both disks on the slow system take about
102s to write 4GB to disk, or around 41.3GB/s.  That isn't horrible
horrible, but it isn't great either.  On the faster machine the WD10EARS
does the job in 39 seconds, and even the old Seagate is done in 74s.  It
strikes me that something must be rate limiting both disks to about the
same throughput.  The Sil 3114 chip is somehow interfaced through the
PCI bus, but even if that is only 33MHz it is still 4 bytes wide and
should be able to handle around 132 MB/s, 3X what I'm seeing.  All of
the PCI and PCI-X slots are unoccupied. I have no previous experience
with the Sil 3114 or the Arima board, so don't know if this is typical
for either.

Perhaps the oddest part of this is that during these tests the disk
light on the slow system blinks but is often off for long periods. 
Conversely, on the faster system the disk light stays on pretty
steadily.  As if on the slower system it is doing something else when it
should be doing disk IO.

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech