[Beowulf] mlx 10g ethernet

[Michael Di Domenico]

thanks, most of that i've already tried.  i don't have iperf, but i
did run qperf, which shows 1.0GB/sec and 16us latency between the two
hosts.  but i'm not entirely sure i believe it

the only thing that made a difference so far is changing the rx tx
rings up 8192 on the host.  i went from 200-300MB/sec to 300-400MB/sec
consistently.  but i'm still far short

On Fri, Aug 4, 2017 at 12:47 PM, Alex Chekholko
<alex.chekholko at gmail.com> wrote:
> The ESNet pages are pretty good:
> https://fasterdata.es.net/host-tuning/
> https://fasterdata.es.net/network-tuning/
>
> Since you're on a local low-latency link, all the stuff about high-bandwidth
> high-latency links doesn't apply to you, but the host tuning parameters at
> least give you somewhere to look.
>
> I recommend iperf for testing before IMB, make sure you can get line rate
> throughput with an iperf test.
>
> On Fri, Aug 4, 2017 at 8:52 AM Michael Di Domenico <mdidomenico4 at gmail.com>
> wrote:
>>
>> does anyone have any additional tuning tips beyond the random things
>> on the net for 10g ethernet tuning?  i have two identical servers
>> (Quanta Q71L-4U) attached to a single cisco 10g switch, using
>> identical (Mellanox MNPH28B-XTC) cards.  when i run mpi bandwidth
>> tests between the two cards, i top out at 200-300MB/sec using IMB
>> sendrecv.  if i use lustre lnet selftests, i can crank the max to
>> 500MB/sec, but i can't seem to crest this.
>>
>> i've monkied around with the sysctl params and the cards are in x8
>> slots at 2.5GT/s.  i think they should be capable of driving at
>> ~900MB/sec, but i can't seem to get there.  i know i'm missing
>> something...
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf