[Beowulf] Rsync - checksums

[Stu Midgley]

we use rsync to a zfsonlinux fs with compression + dedup and snapshots for
incrementals.

works well :)

means your most recent copy is a live fs people can access easily.


On Mon, Jun 17, 2019 at 9:44 PM Bill Wichser <bill at princeton.edu> wrote:

> We have moved to a rsync disk backup system, from TSM tape, in order to
> have a DR for our 10 PB GPFS filesystem.  We looked at a lot of options
> but here we are.
>
> md5 checksums take a lot of compute time with huge files and even with
> millions of smaller ones.  The bulk of the time for running rsync is
> spent in computing the source and destination checksums and we'd like to
> alleviate that pain of a cryptographic algorithm.
>
> Googling around, I found no mention of using a technique like this to
> improve rsync performance.  I did find reference to a few hashing
> algorithms though which could certainly work here (xxhash, murmurhash,
> sbox, cityhash64).
>
> Rsync has certainly been around for a few years!  We are going to pursue
> changing the current checksum algorithm and using something much faster.
>   If anyone has done this already and would like to share their
> experiences that would be wonderful. Ideally this could be some optional
> plugin for rsync where users could choose which checksummer to use.
>
> Bill
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>


-- 
Dr Stuart Midgley
sdm900 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20190618/17156d90/attachment-0001.html>