[Beowulf] mlx 10g ethernet

[Michael Di Domenico]

found a copy of iperf on my system, so i ran it.  show 0-10sec
10.6GBytes/sec 9.08Gbit/sec

so given that qperf seems to agree with iperf, i guess it's an
interesting question now why, lustre lnet_selftest and IMB sendrecv
seem throttled at 500MB/sec

On Fri, Aug 4, 2017 at 1:21 PM, Michael Di Domenico
<mdidomenico4 at gmail.com> wrote:
> thanks, most of that i've already tried.  i don't have iperf, but i
> did run qperf, which shows 1.0GB/sec and 16us latency between the two
> hosts.  but i'm not entirely sure i believe it
>
> the only thing that made a difference so far is changing the rx tx
> rings up 8192 on the host.  i went from 200-300MB/sec to 300-400MB/sec
> consistently.  but i'm still far short
>
> On Fri, Aug 4, 2017 at 12:47 PM, Alex Chekholko
> <alex.chekholko at gmail.com> wrote:
>> The ESNet pages are pretty good:
>> https://fasterdata.es.net/host-tuning/
>> https://fasterdata.es.net/network-tuning/
>>
>> Since you're on a local low-latency link, all the stuff about high-bandwidth
>> high-latency links doesn't apply to you, but the host tuning parameters at
>> least give you somewhere to look.
>>
>> I recommend iperf for testing before IMB, make sure you can get line rate
>> throughput with an iperf test.
>>
>> On Fri, Aug 4, 2017 at 8:52 AM Michael Di Domenico <mdidomenico4 at gmail.com>
>> wrote:
>>>
>>> does anyone have any additional tuning tips beyond the random things
>>> on the net for 10g ethernet tuning?  i have two identical servers
>>> (Quanta Q71L-4U) attached to a single cisco 10g switch, using
>>> identical (Mellanox MNPH28B-XTC) cards.  when i run mpi bandwidth
>>> tests between the two cards, i top out at 200-300MB/sec using IMB
>>> sendrecv.  if i use lustre lnet selftests, i can crank the max to
>>> 500MB/sec, but i can't seem to crest this.
>>>
>>> i've monkied around with the sysctl params and the cards are in x8
>>> slots at 2.5GT/s.  i think they should be capable of driving at
>>> ~900MB/sec, but i can't seem to get there.  i know i'm missing
>>> something...
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf