[Beowulf] OT: recoverable optical media archive format?

[Thanassis Tsiodras]

The corruption of rsbep-protected data with GCC versions later than
4.3 has now been fixed.

I reviewed the code with valgrind, and it turned out that the code I
"inherited" from the original rsbep package performed out of bounds
read accesses in the "distribute" function... This was not in the
"core" Reed-solomon code, only in the interleaving function. I
re-wrote it - check rsbep.c, line 68 in the latest tarball. I also did
a clean-up of useless code that was never called.

The results work perfectly under all GCC versions I tried, under
Debian/32, Arch/32 and FreeBSD/64. Give it a try:
http://users.softlab.ntua.gr/~ttsiod/rsbep-0.1.0-ttsiodras.tar.bz2

So, to summarize, my response points on the original thread:

- The errors reported by David Mathog on the list had to do with
erroneous usage of the tools -  either the "freeze"/"melt" scripts
must be used, or "chopping" of the output has to be done via your own
custom code.
- The memory read access errors were fixed, so the tool works fine
with all GCC versions under all OSes.
- David's "pockmark" app is not representative of what happens in
storage media - they don't fail on byte-boundaries - they fail on
sector boundaries. My small contributions (freeze/melt scripts) on
rsbep make sure that even if we lose 127 contiguous 512-byte sectors,
we can still recover the data, at the exact original size.
- If you want bullet-proof checks, you can easily add the MD5 or SHA
sum of the input data, to the "to-be-shielded-stream", so that the
"melting" can be 100% certain of successful in detecting successful
restoration or data. This, however, is not necessary if you use an
algorithm that can detect errors in the decoded stream (gzip, bzip2,
etc)
- One final comment about PAR, which was suggested by others: since it
was designed for newsgroups, its recovery capabilities had other
(non-storage related) scope - for an "executive summary" read the last
of the comments I received when my rsbep became Slashdotted, here:
http://hardware.slashdot.org/hardware/08/08/03/197254.shtml

-- 
What I gave, I have; what I spent, I had; what I kept, I lost. -Old Epitaph