[Beowulf] OT: recoverable optical media archive format?

[Michael Di Domenico]

What's the ramification of losing a block?  (ie file-system won't
mount, data has a hole)

Not that it's elegant, the first thing that pops to mind is using
'split' to chunk the file into many little bits and then md5 each bit


On Tue, Jun 8, 2010 at 1:44 PM, David Mathog <mathog at caltech.edu> wrote:
> This is off topic so I will try to keep it short:  is there an
> "archival" format for large binary files which contains enough error
> correction to that all original data may be recovered even if there is a
> little data loss in the storage media?
>
> For my purposes these are disk images, sometimes .tar.gz, other times
> gunzip -c of dd dumps of whole partitions which have been "cleared" by
> filling the empty space with one big file full of zero, and then that
> file deleted.  I'm thinking of putting this information on DVD's (only
> need to keep it for a few years at a time) but I don't trust that media
> not to lose a sector here or there - having watched far too many
> scratched DVD movies with playback problems.
>
> Unlike an SDLT with a bad section, the good parts of a DVD are still
> readable when there is a bad block (using dd or ddrescue) but of course
> even a single missing chunk makes it impossible to decompress a .gz file
> correctly.  So what I'm looking for is some sort of .img.gz.ecc format,
> where the .ecc puts in enough redundant information to recover the
> underlying img.gz even when sectors or data are missing.   If no such
> tool/format exists then two copies should be enough to recover all of an
> .img.gz so long as the same data wasn't lost on both media, and if bad
> DVD sectors always come back as "failed read", never ever showing up as
> a good read but actually containing bad data.  Perhaps the frame
> checksum on a DVD is enough to guarantee that?
>
> Thanks,
>
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>