[Beowulf] Whats up with these newer Intel NICs?

[Joe Landman]

Jeff Johnson wrote:
> Joe,
> 
>    I think you may be dealing with a PCIe fifo issue.

Hi Jeff:

  Possibly.  I had thought about that.  I was thinking more along the
lines of "it is a motherboard NIC, so we don't need no steenkeen high
performance things like 64 bit buffers ..."

> 
>    I have seen issues with the Intel PCIe gigabit ethernet onboard parts
> when compared to PCIe slot cards and PCIX cards like the ones you are
> testing. Specifically the partitioning of the controller's buffers
> between rcv and xmit operations (internal to the controller chip itself)
> and the controller's relationship with the PCIe buffer on the
> northbridge. PCIe, being serial, has different challenges when reaching
> the top end of a device's performance capabilities. In this case you are
> suffering some buffer throttling.

I played with some (OS/NIC) buffer settings, txqueuelen, and a few other
tunables.  Nothing seems to have impacted it.

>    By default the buffers are partitioned for "one size fits most"
> scenario. If you know your i/o profile you can use ethtool (or modify

Yeah ...

> e1000 driver source) to repartition the controller's fifo to favor rcv
> or xmit operations. This results in better performance in situations
> where you know you will have heavier writes over reads or vice versa.

Of course, though without knowing your workload in advance you can't
really tune this.

Aside from that, I can't say I have seen many people tune their storage
clusters for workloads of one particular type.  You basically never know
what users will throw your way, and you really don't want one "corner
case" test being the important thing that drives down overall performance.

> 
>    *OR* it is because you are using a Supermicro motherboard..  =)

Owie ... that left a mark ...

I thought it was that I hadn't given the appropriate HPC deity their
burnt (processor) offering ...



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
       http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615