Intel 860 PCI bandwidth problem

[Steffen Persvold]

Maurice Hilarius wrote:
> 
> In recent test on motherboards with Intel 860 chipsets we were seeing less
> than wonderful transfer rates using Wulfkit and Myrinet cards.
> 
> After some explorations on kernel issues, and other hardware forums we were
> still not seeing any reason why this was happening.
> 
> Recently Intel published updated chipset errata lists, and I scanned over them.
> 
> One issue quickly popped out at me, and I now know what the problem seems
> to be:
> In the file found at:
> ftp://download.intel.com/design/chipsets/specupdt/29071501.pdf
> 
> Intel lists errata for the 860 chipset.
> One of these states:
> "5. Sustained PCI Bandwidth Problem:
>   During a memory read multiple operation, a PCI master will read more than
> one complete cache line from memory. In this situation, the MCH pre-fetches
> information from memory to provide optimal performance. However, the MCH
> cannot provide information to the PCI master fast enough. Therefore, the
> ICH2 terminates the read cycle early to free up the PCI bus for other PCI
> masters to claim.
> 
> Implication: The early termination limits the maximum bandwidth to ~90 MB/s.
> 
> Workaround: None
> 
> Status: Intel has no fix planned for this erratum."
> 
> This effectively eliminates the 860 chipset motherboards from contention
> for HPTC clustering use, IMHO.
> 
> Any thoughts from anyone on this?
> 

This only affects DMA operations which uses the PCI instruction "Read multiple". Normal Wulfkit
usage (ScaMPI) is with PIO and is therefore not affected by this issue. Instead, PIO performance on
these chipsets is limited by the fact that we cannot get more than 32byte burst (yet), giving you
approx 170MByte/sec.

Regards,
-- 
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best   
 mailto:sp at scali.no  |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.12.2 -         
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >300MBytes/s and <4uS latencyy