[Beowulf] 10G and rsync

[David Mathog]

On Thu, 2 Jan 2020 11:27:58 -0500 Michael Di Domenico wrote
>> 2) how many files are you xfering?  Are these generally large files or
>> many small files, or a distribution with a long tail towards small
>> files?
> 
> small, 100-200 files, definitely not a MDT issue

1. Is a single large file transfer rate reasonable?

2. Ditto for several large files?

3. If you examine the transfer log for your 100-200 files and then feed 
those file names into tar to produce a local tar file is that rate 
reasonable?

4. Transfer of that single tar file reasonable?  (Presumably yes if 1. 
was OK.)

5. If you untar that single file on the remote system, is the rate 
reasonable?

What I'm getting at is that on one end or the other the transfer may be 
overwhelming the disk/RAID buffers and once the physical disk heads have 
to bounce around between different parts of the disk the rate goes down 
the tubes.

Are you sure there is not a patrol read ongoing on one system or the 
other?  That can cause this sort of disk head issue.

Also it might be this "hugepage" issue:

https://www.beowulf.org/pipermail/beowulf/2015-July/033282.html

The fix for that was:

      echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech