Testing the GNIC-II (Hamachi) gigabit NIC
Paul Farrell
farrell@mcs.kent.edu
Wed Nov 18 21:48:47 1998
In the context of the recent discussion of increased MTU,
although increasing MTU would be of considerable benefit, tuning other
parameters has a considerable effect on the performance achievable.
In particular, changing the TCP send and receive socket sizes can have
a substantial effect. Also of importance, if you are using Linux,
is which version of the Linux kernel is used. The latest development kernel
(at the start of our testing 2.1.121), has considerably better TCP/IP
drivers than the stable kernel (2.0.35). Among other things these support
RFC 1323 window scaling etc, which I believe is not supported in the stable
kernel. This was manifested not just in peak performance but in the general
graph of performance v message size. In the case of the stable kernel this is
very jagged, whereas for the developmental kernel it is essentially
monotonic increasing with the exception of one dip in performance.
We have conducted tests of TCP performance on a number of platforms
using netperf and netpipe. In later tests the netpipe was a modified
version where read/writes were replaced by send/recv.
In particular initial tests were on 2 Dell Pentium II/350s:
Dell Optiplex GX-1 Intel Pentium II - 350MHz
33MHz/32Bit PCI Slots
GNIC-II Gbs Ethernet Card (Preproduction)
Redhat Linux 5.1 with 2.0.35 (the latest stable kernel) and
2.1.121 (which was the current developmental kernel).
Later tests also used a Celeron 300 MHz with 66 MHz bus overclocked
to 100MHz bus giving an effective 450 MHz processor, and a Dell
Pentium II/450.
Kernel version:
On the 2 Dell Pentium II/350s the initial tests were with
netpipe default test (roundtrip) - note that this is not the stream
test performed as the default by netperf, which gives better performance.
For the stable kernel the peak performance was about 226 Mbps for messages
of 196611 bytes, whereas for the developmental kernel it was 322 Mbps for
very large messages and above 300 Mbps for messages above 131069 bytes.
All later tests used the developmental kernel.
Processor speed:
Testing between the Celeron and Pentium II/450 increased this
peak performance to 348 Mbps.
Ethernet rx/tx buffer:
The Hamachi has a default rx/tx of 64K/32K
#define TX_RING_SIZE 16
#define RX_RING_SIZE 32
Changing this to
#define TX_RING_SIZE 32
#define RX_RING_SIZE 128
The effect of changing the transmit ring buffer (tx)
and receive ring buffer (rx) in the Hamachi gigabit ethernet driver,
is uncertain, but for the Pentium II/350 seems irrelevant for values of
rx=32,64,128 and tx=32 but decreases performance for rx=256,tx=32.
Later we standardized on rx=64K, tx=32K.
Socket size:
Changing the socket size from 64K (which is really 65535 bytes in the
linux driver) to 65000 bytes actually increased peak performance to
369 Mbps on the 2 Pentium II/350s.
How performance is measured and sockets again:
What is more dramatic in terms of the attainable performance is
changing to the netperf default test of TCP_STREAM or the equivalent
netpipe stream test (-s). These give substantially higher numbers.
>From a Pentium II/350 to overclocked Celeron the default netpipe
test gives 348 Mbps but the stream test (with no delay set) gives 375 Mbps.
Increasing the send & receive socket buffers to 131070 (the maximum
in the development driver) increases peak performance by the
netpipe stream test to 503 Mbps for 12800 byte messages, and it
remains mostly above 470 Mbps for messages up to 12583424 bytes.
The netperf tests with no delay option set are given below.
To summarize increasing the socket size from 65535 to 131070
increases the peak performance from 391.84 to 517.19 Mbps (64.65 MBps).
It seems likely that, if it were possible, increasing the socket buffer
size further would increase performance further.
TCP STREAM TEST to brigit.gb : nodelay
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
65535 65535 8192 10.00 376.47
65535 65535 16384 10.00 376.29
65535 65535 32768 10.00 379.57
65535 65535 65534 10.00 391.84
65535 65535 65535 10.00 376.37
65535 65535 65536 10.00 380.96
65535 65535 131072 9.99 382.52
65535 65535 262144 9.99 381.03
65535 65535 524288 10.00 378.55
65535 65535 1048576 9.99 377.97
65535 65535 2097152 9.99 380.83
65535 65535 2097152 59.99 381.54
TCP STREAM TEST to brigit.gb : nodelay
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
131070 131070 8192 10.00 497.64
131070 131070 16384 9.99 507.53
131070 131070 32768 9.99 506.76
131070 131070 65335 9.99 492.54
131070 131070 65336 9.99 504.24
131070 131070 131070 9.99 501.91
131070 131070 131072 10.00 510.03
131070 131070 262144 10.00 510.30
131070 131070 524288 26.19 46.88
131070 131070 524288 10.00 516.24
131070 131070 1048576 304.48 6.40
TCP STREAM TEST to brigit.gb : nodelay
netperf: receive_response: no response received. errno 2 counter 0
131070 131070 1048576 10.00 517.19
131070 131070 1048576 9.99 498.37
131070 131070 2097152 98.22 0.01
131070 131070 2097152 10.00 516.26
131070 131070 2097152 10.00 513.16
131070 131070 50331651 10.00 503.82
131070 131070 50331651 9.99 500.29
The anomalous results (where time is much greater than 10 sec)
and the error seem to result from the send blocking.
Paul Farrell
Hong Ong
Roy Heath
---------------------------------------------------------------------
Paul A. Farrell farrell@mcs.kent.edu
Professor, Computer Science Phone: (330) 672-4004 ext 258
Department of Mathematics & Fax: (330) 672-7824
Computer Science Dept: (330) 672-4004
Kent State University
Kent, OH 44242, U.S.A.
---------------------------------------------------------------------
| To unsubscribe, send mail to Majordomo@cesdis.gsfc.nasa.gov, and within the
| body of the mail, include only the text:
| unsubscribe this-list-name youraddress@wherever.org
| You will be unsubscribed as speedily as possible.