[Beowulf] Re: typical latencies for gigabit ethernet
atchley at myri.com
Mon Jun 29 07:26:05 PDT 2009
On Jun 29, 2009, at 5:59 AM, Dave Love wrote:
>> Can you say something about any tuning you did to get decent results?
> To get the lowest latency, turn off rx interrupt coalescence, either
> with ethtool or module parameters, depending on the driver. Of
> you may not want to turn it off completely, depending on how much load
> the extra interrupts cause.
When I test Open-MX, I turn interrupt coalescing off. I run
omx_pingpong to determine the lowest latency (LL). If the NIC's driver
allows one to specify the interrupt value, I set it to LL-1. If the
driver does not allow specifying the actual rate (i.e. it only has
predetermined values), then I leave it off.
The downside is lower throughput for large messages on 10G Ethernet. I
don't think it matters on gigabit.
Brice and Nathalie have a paper which implements an adaptive interrupt
coalescing so that you do not have to manually tune anything:
"Finding a Tradeoff between Host Interrupt Load and MPI Latency over
Ethernet" available at:
I do not know if it has been included in a release yet.
More information about the Beowulf