[Beowulf] Re: typical latencies for gigabit ethernet

Dave Love d.love at liverpool.ac.uk
Mon Jun 29 09:10:03 PDT 2009

Scott Atchley <atchley at myri.com> writes:

> When I test Open-MX, I turn interrupt coalescing off. I run  
> omx_pingpong to determine the lowest latency (LL). If the NIC's driver  
> allows one to specify the interrupt value, I set it to LL-1.

Right, and that's what I did before, with sensible results I thought.
Repeating it now on Centos 5.2 and OpenSuSE 10.3, it doesn't behave
sensibly, and I don't know what's different from the previous SuSE
results apart, probably, from the minor kernel version.  If I set
rx-frames=0, I see this:

rx-usec    latency (µs)
20         34.6
12         26.3
6          20.0
1          14.8

whereas if I just set rx-frames=1, I get 14.7 µs, roughly independently
of rx-usec.  (Those figures are probably ±∼0.2µs.)

> If the  
> driver does not allow specifying the actual rate (i.e. it only has  
> predetermined values), then I leave it off.

Right.  (Adaptive coalescence gave significantly higher latency with our
nVidia and Intel NICs.)

For others interested, this affects TCP results similarly to open-mx,
though the base TCP latency is substantially worse, of course.

I was going to write this up for the OMX FAQ, but was loath to without
understanding the tg3 situation.

> The downside is lower throughput for large messages on 10G Ethernet. I  
> don't think it matters on gigabit.

It doesn't affect the ping-pong throughput significantly, but I don't
know if it has any effect on the system overall (other cores servicing
the interrupts) on `typical' jobs.

> Brice and Nathalie have a paper which implements an adaptive interrupt  
> coalescing so that you do not have to manually tune anything:

Isn't that only relevant if you control the firmware?  I previously
didn't really care about free firmware for devices in the same way as
free software generally, but am beginning to see reasons to care.

More information about the Beowulf mailing list