[Beowulf] Upgrading to gigabit

[Jeffrey B. Layton]

Joe Landman wrote:

> Jeffrey B. Layton wrote:
>
>>   How many servers do you have? The 32-bit, PCI, Intel GigE
>> NICs are pretty good and inexpensive (less than $30). Plus you
>> could use something like an SMC 8505T (5-port) or 8508T
>> (8-port) GigE switch that will give you jumbo frames, for a
>> pretty low cost (the 8508T is about $100).
>
> Interestingly, the SMC and similar low end gigabit cards seem to flood 
> the PC's with interrupts under heavy load.   I've had problems with 
> the tg3 driver in the past on Broadcomm units.  The bcm5700 driver 
> seemed to fix these, though I thought I heard that this driver was 
> being set aside in favor of tg3.


I heard the tg3 driver was awful. I don't have any experience with it
though. Doug and I have had pretty good luck with the Intel NICs
(we're using them in Kronos). They are pretty cheap right now.
However, we did have about a 25% failure rate with them (2 out of
8). We ended up turining off Interrupt Throttling since this was
hurting hpl performance (as well as NPB). When it was turned
on, it gave great latency. Even when it turned off, we're still seeing
Netpipe latencies of about 28 us. Not bad for GigE and inexpensive
switches.