[Beowulf] OT: recoverable optical media archive format?

[Jesse Becker]

I came across this page a few years back that discusses this very
problem:

    http://users.softlab.ntua.gr/~ttsiod/rsbep.html



On Tue, Jun 08, 2010 at 01:44:55PM -0400, David Mathog wrote:
>This is off topic so I will try to keep it short:  is there an
>"archival" format for large binary files which contains enough error
>correction to that all original data may be recovered even if there is a
>little data loss in the storage media?  
>
>For my purposes these are disk images, sometimes .tar.gz, other times
>gunzip -c of dd dumps of whole partitions which have been "cleared" by
>filling the empty space with one big file full of zero, and then that
>file deleted.  I'm thinking of putting this information on DVD's (only
>need to keep it for a few years at a time) but I don't trust that media
>not to lose a sector here or there - having watched far too many
>scratched DVD movies with playback problems.
>
>Unlike an SDLT with a bad section, the good parts of a DVD are still
>readable when there is a bad block (using dd or ddrescue) but of course
>even a single missing chunk makes it impossible to decompress a .gz file
>correctly.  So what I'm looking for is some sort of .img.gz.ecc format,
>where the .ecc puts in enough redundant information to recover the
>underlying img.gz even when sectors or data are missing.   If no such
>tool/format exists then two copies should be enough to recover all of an
>.img.gz so long as the same data wasn't lost on both media, and if bad
>DVD sectors always come back as "failed read", never ever showing up as
>a good read but actually containing bad data.  Perhaps the frame
>checksum on a DVD is enough to guarantee that?
>
>Thanks,
>
>David Mathog
>mathog at caltech.edu
>Manager, Sequence Analysis Facility, Biology Division, Caltech
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Jesse Becker
NHGRI Linux support (Digicon Contractor)