[Beowulf] dedupe filesystem
lindahl at pbm.com
Mon Jun 8 13:52:48 PDT 2009
>> It might be worth noting that dedup is not intended for high
>> performance file systems ... the cost of computing the hash(es)
>> is(are) huge.
> Some file systems do (or claim to do) checksumming for data integrity
> purposes, this seems to me like the perfect place to add the computation
> of a hash - with data in cache (needed for checksumming anyay), the
> computation should be fast.
Filesystems may call it a "checksum" but it's usually a hash. We use a
Jenkins hash, which is fast and a lot better than, say, the TCP
checksum. But it's a lot weaker than an expensive hash.
If your dedup is going to fall back to byte-by-byte comparisons, it could
be that a weak hash would be good enough.
More information about the Beowulf