[Beowulf] Home beowulf - NIC latencies
diep at xs4all.nl
Fri Feb 4 13:33:47 PST 2005
Thanks for your deep inside, this is very helpful!
At 12:57 4-2-2005 -0700, Josip Loncaric wrote:
>Vincent Diepeveen wrote:
>> At 00:29 4-2-2005 -0800, Bill Broadley wrote:
>>>Do you know that gigabit is too high latency?
>Gigabit Ethernet adapters often need tweaking to deliver reasonable
>latency, bandwidth, and CPU utilization.
>For example, if your system uses the e1000 driver (Intel's gigabit
>Ethernet), the default setting is "dynamic Interrupt Throttle Rate" --
>which means that the card will delay interrupting the CPU by up to about
>130 microseconds after receiving a packet. Moreover, the "dynamic" part
>causes the network chip microcode to vary this delay in multiples of
>about 16 microseconds, so that different packets will generally
>experience different receive delays.
>For the e1000 driver,
>recommends using "options e1000 InterruptThrottleRate=80000" (add this
>line to /etc/modules.conf). Users of this driver may also want to check
>Intel's parameters for e1000 listed at
>http://www.intel.com/support/network/sb/cs-009209.htm#parameters -- just
>don't assume that the default values are appropriate for cluster use.
>Other gigabit Ethernet adapters have similar interrupt mitigation
>strategies, all designed to gracefully cope with high packet rates at
>high network speeds. For cluster use, adjustments are usually advisable.
>The basic Rx interrupt mitigation scheme is this: the receiver's CPU
>won't be interrupted until at least N packets have arrived or M
>microseconds have elapsed (whichever comes first). This clearly adds up
>to M microseconds to network latency. BTW, one often sees N=6
>(otherwise NFS performance can seriously degrade) and M>=16. Other
>variants of this basic scheme are possible; but they all mean increased
>Finally, don't forget the Tx side interrupt mitigation, or else the
>sending CPU might not be told promptly that it's OK to send more. The
>default Tx settings are probably fine for full size packets, but if your
>applications send lots of small packets, tweaking your network driver's
>Tx settings may help.
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf