[Beowulf] HD undetectable errors
Nifty Tom Mitchell
niftyompi at niftyegg.com
Tue Aug 25 23:44:24 PDT 2009
On Wed, Aug 26, 2009 at 12:18:17AM +0200, Reuti wrote:
> Am 25.08.2009 um 22:37 schrieb Nifty Tom Mitchell:
>> However, It does make sense to me to generate and maintain site
>> specific meta
>> data for all valuable data files to include both detection (yes tamper
>> detection too) and recovery codes.
> Sounds like using local par or par2 files along with their hash
> information about the original files. Maybe this could be implemented as
> FUSE filesystem for easy handling (which will automatically split the
> files, create the hashes and any number of par files you like).
> -- Reuti
Interesting... par2 looks close.
It may solve my worries with Cloud storage.
The differences between par and par2 are interesting. The various
issues involving damaged files (even a single bit error) in the
initial design of par were a limitation.
I can see that there has been a lot of work done already
making it is a good place to start. And yes slipping this
under a FUSE filesystem might hide the pain for local storage.
>> I would extend this to all data with
>> the hope that any problems might be seen first on inconsequential
>> Tripwire might be a good model for starting out on this.
>> I should note that the three 'big' error rate problems I have worked on
>> in the past 25 years had their root cause in an issue not understood
>> or considered at design time so empirical data from the customer was
>> critical. Data sheets and design document conclusions just missed the
>> These experiences taught me to be cautious with storage statistics.
>> Looming in the dark clouds is a need for owning your own data
>> It seems obvious to me in the growing business of cloud computing and
>> cloud storage
>> that you need to "trust but verify" the integrity of your data. My
>> on this is that external integrity methods are critical in the future.
>> And do remember that "parity is for farmers."
>> T o m M i t c h e l l
>> Found me a new hat, now what?
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>> To change your subscription (digest mode or unsubscribe) visit
T o m M i t c h e l l
Found me a new hat, now what?
More information about the Beowulf