[Beowulf] Performance characterising a HPC application

[Douglas Eadline]

>> You really need two networks for GAMMA as GAMMA turns off TCP
>> while in use. Not to difficult with today's GigE-full nodes.
>> Just need extra an extra switch.
>
> I read the gamma code a long time ago, but had the impression
> it simply used a different 'protocol' field in the IP header.
> if that's correct, it would work fine on the same switch if
> you used different nics.  or do you mean that there would be
> arp issues with both interfaces on the same switch?

Oh, I should have said extra ports (or another switch).
>From the GAMMA page:

"The GAMMA driver is unable to manage standard IP traffic. Therefore, all
IP services must be supported by an additional LAN (an additional NIC on
each cluster node, plus an additional hub/switch)."

>
>>> I just got a X2100 M2 machine in today. 4 GBit ports, though 2 of them
>>> nVidia.
>>
>> I was able to get 17 usecond latency (netpipe TCP ) with the nvidia
>> links on these systems. Out of the box broadcom was about
>> 40 useconds (netpipe TCP).
>
> wow, impressive.  I've never really had a satisfying explanation of why
> gigabit tends to result in such high latencies.  (I wouldn't call 17
> high!)

I was surprised as well. The X2100 nodes have both a broadcom and nvidia
GigE port. I "assumed" broadcom would be better, but just in case, I ran
netpipe/TCP between two nodes. BTW, my previous netpipte/TCP best was
29 us (Intel 8254x chipset family, with the InterruptThrottleRate
turned off, but be aware this is not optimal for all applications)

As I tend to play with the "low end value hardware" I am going to
be testing some Core-duo machines with GAMMA very soon. I'll be using
Intel PCIe NICs, which "should" be able to get data off the
motherboard faster than a 32/33 PCI slot (my best was
about 69 MB/sec with 11 us latency using netpipe/MPI using
GAMMA MPI, which is not bad considering it was a $30 NIC)

There is some data on ClusterMonkey as well (search
for e1000 or gamma) http://www.clustermonkey.net


--
Doug



>
> !DSPAM:46056fb814812020149523!
>


-- 
Doug