3x100Mbps: summary

[beowulf at seinfeld.engr.ucf.edu]

On Sat, 7 Oct 2000, Laurent Itti wrote:

> Hi all -
> 
> - performance:
>   * check out netperf.org for a test suite and test results.
>   * a single board yields around 94 Mbps throughput
>   * bonded dual-board setups (with another brand for the boards) have
>     been reported to yield around 187 Mbps throughput
>   * with 3 boards we start seeing diminishing returns but still
>     significantly improved performance over 2 boards:
> 
>   setup: Abit SE6(i815), PIII-733EB, 256MB PC133, 30GB UATA/100, 3xRTL8139
>   linux: Mandrake 7.2beta3 (sept 22, 2000), 2.2.17mdk kernel,
>          after experimentation finally it seems that the rtl8139 driver
>          is more stable than the 8139too driver (both included in
>          the distribution; edit /etc/modules.conf to choose; we can
>          still crash the drivers by running the tests below for very
>          long periods...)
>   state: normal boot and runlevel, lots of processes running, i.e.,
>          "real-life" performance test.  Node n11 is running "netserver".
> 
> [root at n01 /root]# netperf -l 60 -H n11
> TCP STREAM TEST to n11
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>  
>  65535  65535  65535    60.00     244.06
> 
> 
> [root at n01 /root]# netperf -H n11 -t UDP_STREAM -- -m 10240
> UDP UNIDIRECTIONAL SEND TEST to n11
> Socket  Message  Elapsed      Messages
> Size    Size     Time         Okay Errors   Throughput
> bytes   bytes    secs            #      #   10^6bits/sec
> 
>  65535   10240   9.99        32503      0     266.49
>  65535           9.99        32319            264.98
> 
> 
> we can get up to 280 Mbps UDP send (first line) but then reduce the UDP
> receive (second line), by playing with message size. I'll have to read
> netperf's doc to see what that means ;-)

Unfortunately I don't get very good performance when using 3 bonded
channels.  The results I get for 3 channels is rather dismal.

For UDP_STREAM using ./netperf -H node3 -t UDP_STREAM -- -m 10240

UDP UNIDIRECTIONAL SEND TEST to node3
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

 65535   10240   9.99        34860      0     285.79
 65535           9.99         8635             70.79

I have tryed playing around with the message size but still can't get the
receive numbers anywhere near the send.  Could anyone explain this?


For TCP_STREAM using ./netperf -n 2 -l 30 -H node3 -t TCP_STREAM -- -m
10240

TCP STREAM TEST to node3
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 65535  65535  10240    29.99     138.74

This number is worse then the 2 channel TCP_STREAM test.

------------------------------------------------------------------------

When I use 2 bonded channels using the same commands I get

UDP UNIDIRECTIONAL SEND TEST to bnode0
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

 65535   10240   10.00       23450      0     192.20
 65535           10.00       23443            192.14

TCP STREAM TEST to bnode0
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 65535  65535  10240    29.99     157.57

----------------------------------------------------------------------

And for a single channel

UDP UNIDIRECTIONAL SEND TEST to node0
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

 65535   10240   10.00       11729      0      96.12
 65535           10.00       11729             96.12

TCP STREAM TEST to node0
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 65535  65535  10240    29.99      94.06

----------------------------------------------------------------------

We have 16 compute nodes and one server/frontend.

Our setup is:
  Motherboard:	SuperMicro P6DGE Intel BX
  Processor(s):	Dual PII-350
  Memory:	256MB PC100
  HardDrive:	FUJITSU 8GB
  Ethernet:	3 x Kingston KNE100TX (tulip)
  OS:		RedHat 6.1
  Kernel:	2.2.17

Does anyone know why the 3 channel network would perform worse then the 2
channel network?  The only real differences between our setups is that we
use SMP nodes, but you have faster processors.  I know that running in SMP
under linux causes the network latency to increase, but could it be the
cause of our 3 channel network problems?

Judd Tracy
jtr at isl.engr.ucf.edu