[Beowulf] 10G and rsync

[Joe Landman]

On 1/2/20 10:39 AM, Michael Di Domenico wrote:
> On Thu, Jan 2, 2020 at 10:35 AM Chris Dagdigian <dag at sonsorol.org> wrote:
>> - I noticed you did not test small file / metadata operations. My past
>> experience has found that this was the #1 cause of slowness in rsync and
>> other file transfers. iperf and IOR tests are all well and good but you
>> should run something like MDTEST to hammer the system on metadata and
>> small file handling. If you are moving lots of tiny files or hundreds of
>> thousands of directories etc this could be your problem
>> - Single stream over 10gig has never been great for me doing big data
>> movement. I get way more throughput by using rsync in parallel to
>> migrate multi-stream either from a single 10gig connected host or a
>> cluster of them
> thanks.  this is definitely not a metadata issue, i'm only moving
> 100-200 files in total.  this is strictly a single stream transfer
> problem with rsync
>
> if i parallel the rsync, i can in fact increase the performance, but
> the process that's doing the transfer (part of a bigger system) can't
> handle that.
>
> it just seemed unfathomable to me that rsync can't transfer at wire
> past 1G speeds..

Hrm ... at home, with my little arista 10GbE switch and two machines, I 
regularly hit about 180 MB/s between machines (with SATA drives) for 
large xfers with rsync.  This is about the actual read/write speed of 
the drives in this case.

Have you measured this?  Your slowest portion of the chain is likely to 
be related to your bottleneck.  Disk reads often can be that, especially 
if you are sharing actively used disks on one side or the other...



> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

-- 
Joe Landman
e: joe.landman at gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman