[Beowulf] Rsync - checksums

Ellis H. Wilson III ellis at ellisv3.com
Tue Jun 18 08:16:30 PDT 2019

On 6/18/19 11:00 AM, Bill Wichser wrote:
> Well thanks for THAT pointer!  Using --checksum-choice=none results in 
> speedup of somewhere between 2-3 times.  That's my validation of the 
> checksum theory things have been pointing towards.  Now to get xxhash 
> into rsync and I think we are all set.

Glad to hear that improved things for you.  I will caution against using 
a non-crypto hash for this though.  Unless you can verify xxhash offers 
at least as much protection against collisions as MD5, you may not be 
protecting your transfers sufficiently.  I haven't looked into xxhash 
enough to offer advice on this.

If you're already using rsync as just a piece of a larger framework, it 
really might be worth cooking up a script to md5sum the list in 
files-from on both sides in parallel to use all of your cores on both 
sides and then compare the results after the transfer.  Then you can 
just use --checksum-choice=none but still know your data made it safely 
to the other side.



Ellis H. Wilson III, Ph.D.

More information about the Beowulf mailing list