PCI-64: how to find
Patrick Geoffray
patrick at myri.com
Tue Feb 26 22:45:32 PST 2002
Hi,
sorry for the delay.
Velocet wrote:
> right. I dont know if using LAM/MPI ends up generating the same sized
> packets, but for a number of the jobs I've run with gromacs under it
> they all seem to be around 700-1000 bytes. Is that big enough to get a win
> out of 64 bit busses?
With IP, the interconnect part of the latency is very small compared to
the IP stack overhead. For messages size <1KB, you should not see big
difference with IP with 64 bit busses.
> The cost differences are almost 2x the price for super cheap cards. If you
> go with quality (intel, higher end Dlinks) its not as big a difference.
> (ie ARKs being $38 for 32bit and $66 for 64bit for low end stuff).
>
> How much latency is due to the stack then? Can this be improved?
On a 100 us latency for example, I would say 80% is in the software and
20% in the hardware. Yes, it can be improved if the IP stack in Linux
would go zero-copy. Linux folks are talking about it, but it won't be in
stable anytime soon.
Another way to avoid the IP overhead is to use a OS-bypass mechanism for
Gigabit Ethernet, like GAMMA or MVIA. However, these layers are
NIC-specific. That's the cost of having tons of hardware suppliers: they
drive the cost down but diversity is a nightmare for software talking
directely to the hardware.
Patrick
----------------------------------------------------------
| Patrick Geoffray, Ph.D. patrick at myri.com
| Myricom, Inc. http://www.myri.com
| Cell: 865-389-8852 685 Emory Valley Rd (B)
| Phone: 865-425-0978 Oak Ridge, TN 37830
----------------------------------------------------------
More information about the Beowulf
mailing list