[Beowulf] How Would You Test Infiniband in New Cluster?

[Sabuj Pattanayek]

Hi,

The OMB package for mvapich2 (mpich2 for IB) has some great programs
that you can use to test to see if your IB network is working
properly. Here are some of my results on our QDR IB network:

% mpiexec -n 2 ./osu_latency
# OSU MPI Latency Test v3.1.1
# Size            Latency (us)
0                         1.87
1                         1.95
2                         1.97
4                         1.98
8                         1.98
16                        1.99
32                        2.04
64                        2.19
128                       3.63
256                       3.91
512                       4.38
1024                      5.25
2048                      6.80
4096                      8.12
8192                     10.94
16384                    16.35
32768                    22.13
65536                    33.28
131072                   55.09
262144                  100.01
524288                  166.54
1048576                 333.60
2097152                 636.91
4194304                1252.71

% mpiexec -n 2 ./osu_bw
# OSU MPI Bandwidth Test v3.1.1
# Size        Bandwidth (MB/s)
1                         1.44
2                         2.79
4                         5.51
8                        11.18
16                       21.17
32                       43.42
64                       82.41
128                     146.97
256                     314.42
512                     564.18
1024                   1033.38
2048                   1634.33
4096                   2168.96
8192                   2514.58
16384                  2788.07
32768                  3038.48
65536                  3213.89
131072                 3293.78
262144                 3334.07
524288                 3353.60
1048576                3355.25
2097152                3362.15
4194304                3365.81

That's 3.3GB/s or ~26.4gbps .

HTH,
Sabuj Pattanayek

On Tue, Nov 17, 2009 at 12:33 PM, Bill Broadley <bill at cse.ucdavis.edu> wrote:
> Jon Forrest wrote:
>> Let's say you have a brand new cluster with
>> brand new Infiniband hardware, and that
>> you've installed OFED 1.4 and the
>> appropriate drivers for your IB
>> HCAs (i.e. you see ib0 devices
>> on the frontend and all compute nodes).
>> The cluster appears to be working
>> fine but you're not sure about IB.
>>
>> How would you test your IB network
>> to make sure all is well?