PCI-64: how to find

[Patrick Geoffray]

Hi,

sorry for the delay.

Velocet wrote:
> right. I dont know if using LAM/MPI ends up generating the same sized
> packets, but for a number of the jobs I've run with gromacs under it
> they all seem to be around 700-1000 bytes. Is that big enough to get a win
> out of 64 bit busses?

With IP, the interconnect part of the latency is very small compared to 
the IP stack overhead. For messages size <1KB, you should not see big 
difference with IP with 64 bit busses.

> The cost differences are almost 2x the price for super cheap cards. If you
> go with quality (intel, higher end Dlinks) its not as big a difference.
> (ie ARKs being $38 for 32bit and $66 for 64bit for low end stuff).
> 
> How much latency is due to the stack then? Can this be improved?

On a 100 us latency for example, I would say 80% is in the software and 
20% in the hardware. Yes, it can be improved if the IP stack in Linux 
would go zero-copy. Linux folks are talking about it, but it won't be in 
stable anytime soon.
Another way to avoid the IP overhead is to use a OS-bypass mechanism for 
Gigabit Ethernet, like GAMMA or MVIA. However, these layers are 
NIC-specific. That's the cost of having tons of hardware suppliers: they 
drive the cost down but diversity is a nightmare for software talking 
directely to the hardware.

Patrick
----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      patrick at myri.com
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------