[Beowulf] mysterious slow SATA on one machine

[Geoff Jacobs]

David Mathog wrote:
> I'm hoping somebody has seen this before and can suggest what might be
> going on.  
> 
> One machine (Arima HDAMA-I board, dual Opteron 280, 4GB RAM,
> Sil 3114 Sata controller, Sil 5.4.03 firmware) has mysteriously slow
> SATA IO.  This is the case for two different disks (WD10EARS and
> ST340014AS), two different disk schedulers, and two different OS's
> (Mandriva 2010.0 and PLD 2.97 rescue linux.)  Using a different brand of
> cable, and plugging into a different SATA port didn't help either. 
> However, move those disks to another machine (Asus A8N5X, Nvidia CK804
> SATA controller, single core, 1 GB RAM, Knoppix) and they are both much
> faster.  Raw results from various experiments here:
> 
>   http://saf.bio.caltech.edu/pub/pickup/bonnie++.rtf
>   http://saf.bio.caltech.edu/pub/pickup/sustained_write.rtf
> 
> For the sustained write test both disks on the slow system take about
> 102s to write 4GB to disk, or around 41.3GB/s.  That isn't horrible
> horrible, but it isn't great either.  On the faster machine the WD10EARS
> does the job in 39 seconds, and even the old Seagate is done in 74s.  It
> strikes me that something must be rate limiting both disks to about the
> same throughput.  The Sil 3114 chip is somehow interfaced through the
> PCI bus, but even if that is only 33MHz it is still 4 bytes wide and
> should be able to handle around 132 MB/s, 3X what I'm seeing.  All of
> the PCI and PCI-X slots are unoccupied. I have no previous experience
> with the Sil 3114 or the Arima board, so don't know if this is typical
> for either.
> 
> Perhaps the oddest part of this is that during these tests the disk
> light on the slow system blinks but is often off for long periods. 
> Conversely, on the faster system the disk light stays on pretty
> steadily.  As if on the slower system it is doing something else when it
> should be doing disk IO.

As mentioned to David in a separate post, I see similar (worse)
performance deltas using an S-I controller. I see the same delta using
sata_sil driving an ATI SB4xx south bridge.

It might be kernel related, as escalated here:
https://bugzilla.redhat.com/show_bug.cgi?id=502499

-- 
Geoffrey D. Jacobs