Recent AMD760MPX chipset PCI measurements??

[Patrick Geoffray]

Paul Lu wrote:

> Agreed.  Greg Lindahl also pointed this out in a separate reply.
> 
> I was hoping for greater bandwidth to handle the case of a full duplex
> transfer across the Myrinet (which could, in theory, saturate the PCI bus
> given the full 4 Gb/s bandwidth of duplex operation).
> 
> In an email to me, Greg made the point that 4 Gb/s duplex
> is difficult to achieve in practice with the current drivers.
> I haven't tried it, so I defer to Greg (and you) on this issue,
> but I was hoping to have some extra headroom (for duplex transfers
> *and* in scenarios where there is also lots of, say, disk I/O on
> the PCI bus too).


Aargh, Greg beat me here, I was still lost in my endless mail queue.

Ok, briefly: the MPX chipset is, in my opinion a very good one. It's 
certainely not as good as the serverworks one, but it's still very 
attractive for several reasons:
* the PCI Read and PCI Write are well balanced. It's a sign of a good 
internal design. The i860 is very unbalanced for example (PCI Read is 
very very low compare to Write), even with the DT tweak.
* the MPX chipset is on a different segment than the Serverworks: PIII 
are OK, but you want eventually to move to the upper segment that 
includes Athlon and P4. In this segment, where you start to get several 
flops per cycle, the MPX chipset is just the best one so far.
* it provides more than the Myrinet one-way link speed.
* it provides just the bidirectional bandwidth achievable today with 
Myrinet.

Let me develop the last point: The link speed is 2 Gb/s. A full duplex 
link provides 2+2 Gb/s, so if the PCI is big enough you should be able 
to reach the peak of the PCI by doing bidirectional traffic. Everything 
on the board is designed to support 450-500 MB/s bidirectional, at least 
on the C cards (L9). Unfortunaltely, the max bidirectional you get right 
now is 300 MB/s, even if the PCI can sustain much more. My personal 
guess is that the firmware is serializing some events in the state 
machine and the pipeline is damaged. It's on my plate right now and I 
have finished a precise embedded log mechanism today. Hopefully, it will 
tell me which transition in the state machine is late so we can fix it.

So with the current firmware, you won't see a big difference between 
ServerWorks and MPX because they are both more than the one-way link 
speed and because they are better or just enough the bidirectional 
bandwidth. However, I expect to know much more about the bidirectional 
behaviour this week to change that.
And as the Athlon pump more flops than the PIII, MPX based machines are 
more interesting than ServerWorks ones. Now, the plumas chipset from 
Intel and the Grand Champion from ServerWorks are supposed to arrive 
real soon, it may totally change the picture.

And something I like a lot about the MPX design is that the AGP bus is 
bridged from the 64/66 PCI, and the bridged is implemented on both ways: 
it's possible to send/recv to/from the memory on a AGP video card. Some 
cool parallel rendering and video transfert can be done...

Hope it fill the hole.

Patrick

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      patrick at myri.com
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------