[Beowulf] 1.2 us IB latency?

[Patrick Geoffray]

greg.lindahl at qlogic.com wrote:
>> Back then we were struggling with PIO transfers and how they were
>> treated in the CPU/North bridge (write combining and all that). I
>> believe this might still be an issue, correct ?

WC is well implemented on Opteron, it will aggregate consecutive PIO 
writes at 16, 32 and 64 Bytes smoothly. On Intel processors, this is 
more painful: WC is only 64 Bytes. If you flush the WC buffer with less 
than 64 bytes in it, you will see multiple 8-byte PIO writes, and not 
always in order.

> cases we can manipulate the mtrrs after boot to fix this. Getting
> formal support for PAT in the Linux kernel is the long-term fix for
> this.

It's interesting to note that most current OSes have native PAT support, 
except Linux. Even Windows does it well :-)

Patrick