[Beowulf] Whats up with these newer Intel NICs?

Mon Sep 24 15:16:16 PDT 2007

Joe,

    I think you may be dealing with a PCIe fifo issue.

    I have seen issues with the Intel PCIe gigabit ethernet onboard 
parts when compared to PCIe slot cards and PCIX cards like the ones you 
are testing. Specifically the partitioning of the controller's buffers 
between rcv and xmit operations (internal to the controller chip itself) 
and the controller's relationship with the PCIe buffer on the 
northbridge. PCIe, being serial, has different challenges when reaching 
the top end of a device's performance capabilities. In this case you are 
suffering some buffer throttling.

    By default the buffers are partitioned for "one size fits most" 
scenario. If you know your i/o profile you can use ethtool (or modify 
e1000 driver source) to repartition the controller's fifo to favor rcv 
or xmit operations. This results in better performance in situations 
where you know you will have heavier writes over reads or vice versa.

    *OR* it is because you are using a Supermicro motherboard..  =)

--Jeff


landman at scalableinformatics.com wrote:

> Message: 1
> Date: Sun, 23 Sep 2007 22:48:11 -0400
> From: Joe Landman <landman at scalableinformatics.com>
> Subject: [Beowulf] Whats up with these newer Intel NICs?
> To: Beowulf List <beowulf at beowulf.org>
> Message-ID: <46F7256B.6050407 at scalableinformatics.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi folks:
>
>    Working on trying to figure out why the Intel NICs on these 
> motherboards we are working with are slow.  Ok, slow is a relative term. 
>   More along the lines of "not as fast as they could be" specifically 
> relative to a PCI-x 1000/MT adapter we plugged in.
>
>    Scenario is trying to do some load testing.  I have 4 clients, all 
> with the same version of OS, pounding on our server (part of the load 
> test).  Gigabit, server does channel bonding.  Seeing good results.  But 
> .... on the nodes that use this beast:
>
> 04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit 
> Ethernet Controller (Copper) (rev 01)
>          Subsystem: Super Micro Computer Inc Unknown device 0000
>          Flags: bus master, fast devsel, latency 0, IRQ 1274
>          Memory at c8200000 (32-bit, non-prefetchable) [size=128K]
>          I/O ports at 2000 [size=32]
>          Capabilities: [c8] Power Management version 2
>          Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ 
> Queue=0/0 Enable+
>          Capabilities: [e0] Express Endpoint IRQ 0
>          Capabilities: [100] Advanced Error Reporting
>          Capabilities: [140] Device Serial Number f2-72-32-ff-ff-48-30-00
>
> we get ~70-75 MB/s, while plugging a nice little
>
> 05:02.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet 
> Controller (rev 01)
>          Subsystem: Intel Corporation PRO/1000 MT Dual Port Server Adapter
>          Flags: bus master, 66MHz, medium devsel, latency 52, IRQ 28
>          Memory at c8340000 (64-bit, non-prefetchable) [size=128K]
>          Memory at c8300000 (64-bit, non-prefetchable) [size=256K]
>          I/O ports at 3000 [size=64]
>          [virtual] Expansion ROM at c2000000 [disabled] [size=256K]
>          Capabilities: [dc] Power Management version 2
>          Capabilities: [e4] PCI-X non-bridge device
>
> into a PCI-x slot gives us 92-98 MB/s for our load test (IOzone).  It 
> gives more than that, I am optically averaging.
>
> Ok.  So the mystery is *why*.
>
> First I note that the first unit, which is a motherboard NIC, has 
> "32-bit memory" at a particular address, while the second unit, the 
> 1000/MT card in the PCI-x slot has "64-bit memory" at a different address.
>
> Second, and this is counter intuitive, but the motherboard gigabit unit 
> is on PCI-e (x4 at that!)
>
>   
>> [  115.246121] PCI: Setting latency timer of device 0000:04:00.0 to 64
>> [  115.261547] e1000: 0000:04:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x4) 00:30:48:32:72:f2
>> [  115.290791] PM: Adding info for No Bus:eth0
>> [  115.290843] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
>> [  115.290868] ACPI: PCI Interrupt 0000:04:00.1[B] -> GSI 19 (level, low) -> IRQ 19
>> [  115.290882] PCI: Setting latency timer of device 0000:04:00.1 to 64
>> [  115.306461] e1000: 0000:04:00.1: e1000_probe: (PCI Express:2.5Gb/s:Width x4) 00:30:48:32:72:f3
>> [  115.342947] PM: Adding info for No Bus:eth1
>> [  115.342983] e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
>>     
>
> while the PCI-x is on, well, PCI-x.  And it should be slower.
>
>   
>> [  115.608773] e1000: 0000:05:02.0: e1000_probe: (PCI:33MHz:64-bit) 00:04:23:9e:36:ca
>> [  115.636072] PM: Adding info for No Bus:eth2
>> [  115.636105] e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network Connection
>> [  115.636129] ACPI: PCI Interrupt 0000:05:02.1[B] -> GSI 29 (level, low) -> IRQ 29
>> [  115.902030] e1000: 0000:05:02.1: e1000_probe: (PCI:33MHz:64-bit) 00:04:23:9e:36:cb
>> [  115.928619] PM: Adding info for No Bus:eth3
>> [  115.928648] e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network Connection
>> [  115.928687] ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 24 (level, low) -> IRQ 24
>>     
>
> The driver is 7.3.20-k2-NAPI
>
>   
>> [  110.772712] Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
>> [  110.772717] Copyright (c) 1999-2006 Intel Corporation.
>>     
>
> I know 7.6.5 is out, and I installed it on one of the machines, without 
> any impact.
>
> Motherboard is a Supermicro X7DVA-i I think.  I am also seeing this on a 
> different Supermicro motherboard with dual cores.  Same issue/performance.
>
> Any thoughts?
>
>   


-- 
Best Regards,

Jeff Johnson
Vice President
Engineering/Technology
Western Scientific, Inc
jeff.johnson at wsm.com
http://www.wsm.com

5444 Napa Street - San Diego, CA 92110
Tel 800.443.6699  +001.619.220.6580
Fax +001.619.220.6590

"Braccae tuae aperiuntur"