[Beowulf] Rsync - checksums
sdm900 at gmail.com
Mon Sep 30 17:55:28 PDT 2019
That's pretty awesome, are you going to make it available? or push it
If not... how can we get it?
On Tue, Oct 1, 2019 at 1:09 AM Bill Wichser <bill at princeton.edu> wrote:
> Just wanted to circle back on my orginal question. I changed the rsync
> code adding xxhash and we see about a 3x speedup. Good enough since it
> is very close to not using any checksum speedups.
> On 6/17/19 9:43 AM, Bill Wichser wrote:
> > We have moved to a rsync disk backup system, from TSM tape, in order to
> > have a DR for our 10 PB GPFS filesystem. We looked at a lot of options
> > but here we are.
> > md5 checksums take a lot of compute time with huge files and even with
> > millions of smaller ones. The bulk of the time for running rsync is
> > spent in computing the source and destination checksums and we'd like to
> > alleviate that pain of a cryptographic algorithm.
> > Googling around, I found no mention of using a technique like this to
> > improve rsync performance. I did find reference to a few hashing
> > algorithms though which could certainly work here (xxhash, murmurhash,
> > sbox, cityhash64).
> > Rsync has certainly been around for a few years! We are going to pursue
> > changing the current checksum algorithm and using something much faster.
> > If anyone has done this already and would like to share their
> > experiences that would be wonderful. Ideally this could be some optional
> > plugin for rsync where users could choose which checksummer to use.
> > Bill
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
Dr Stuart Midgley
sdm900 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf