[Beowulf] mlx 10g ethernet

[Michael Di Domenico]

does anyone have any additional tuning tips beyond the random things
on the net for 10g ethernet tuning?  i have two identical servers
(Quanta Q71L-4U) attached to a single cisco 10g switch, using
identical (Mellanox MNPH28B-XTC) cards.  when i run mpi bandwidth
tests between the two cards, i top out at 200-300MB/sec using IMB
sendrecv.  if i use lustre lnet selftests, i can crank the max to
500MB/sec, but i can't seem to crest this.

i've monkied around with the sysctl params and the cards are in x8
slots at 2.5GT/s.  i think they should be capable of driving at
~900MB/sec, but i can't seem to get there.  i know i'm missing
something...