[Beowulf] 10G and rsync

[Michael Di Domenico]

> 1) are you sure your traffic is traversing the high bandwidth link?
> Always good to check ...

Yup, only have one on each side, one switch connecting them together.
(ex network engineer, always check the network first, we can't be
trusted) :)

> 2) how many files are you xfering?  Are these generally large files or
> many small files, or a distribution with a long tail towards small
> files?

small, 100-200 files, definitely not a MDT issue

> 3) wire speed xfers are generally the exception unless you are doing
> large sequential single files.   There are tricks you can do to enable
> this, but they are often complex.  You can use the array of
> writers/readers, and leverage parallelism, but you risk invoking
> congestion/pause throttling on your switch.

agreed, if i use IOR or something alike i can drive lustre to
900MB/sec.  i figured i wouldn't get rsync to push that hard, but i
was hoping for better then 100MB/sec

i'm doing this locally on the box, so i'd hoped ssh and it's
compression/encryption factors wouldn't come into play, but i guess
they do some how.

i've been messing about with the '-e ssh -T -c <cipher> -o
Compression=no -x' option to see if i could tease out something
relevant, but that hasn't turned up anything

i'd be happy if i could even break 120MB/sec at this point.

i wonder if there's something odd with the rsync verions, the client
side is redhat 6, i'll see if i can pull in the source and recompile.
maybe there's some enhancement i'm missing