[Beowulf] Rsync - checksums
Ellis H. Wilson III
ellis at ellisv3.com
Tue Jun 18 08:16:30 PDT 2019
On 6/18/19 11:00 AM, Bill Wichser wrote:
> Well thanks for THAT pointer! Using --checksum-choice=none results in
> speedup of somewhere between 2-3 times. That's my validation of the
> checksum theory things have been pointing towards. Now to get xxhash
> into rsync and I think we are all set.
Glad to hear that improved things for you. I will caution against using
a non-crypto hash for this though. Unless you can verify xxhash offers
at least as much protection against collisions as MD5, you may not be
protecting your transfers sufficiently. I haven't looked into xxhash
enough to offer advice on this.
If you're already using rsync as just a piece of a larger framework, it
really might be worth cooking up a script to md5sum the list in
files-from on both sides in parallel to use all of your cores on both
sides and then compare the results after the transfer. Then you can
just use --checksum-choice=none but still know your data made it safely
to the other side.
Ellis H. Wilson III, Ph.D.
More information about the Beowulf