[Beowulf] OT: recoverable optical media archive format?
rtomek at ceti.com.pl
Tue Jun 8 12:19:44 PDT 2010
On Tue, 8 Jun 2010, David Mathog wrote:
> This is off topic so I will try to keep it short: is there an
> "archival" format for large binary files which contains enough error
> correction to that all original data may be recovered even if there is a
> little data loss in the storage media?
> For my purposes these are disk images, sometimes .tar.gz, other times
> gunzip -c of dd dumps of whole partitions which have been "cleared" by
> filling the empty space with one big file full of zero, and then that
> file deleted. I'm thinking of putting this information on DVD's (only
> need to keep it for a few years at a time) but I don't trust that media
> not to lose a sector here or there - having watched far too many
> scratched DVD movies with playback problems.
> Unlike an SDLT with a bad section, the good parts of a DVD are still
> readable when there is a bad block (using dd or ddrescue) but of course
> even a single missing chunk makes it impossible to decompress a .gz file
> correctly. So what I'm looking for is some sort of .img.gz.ecc format,
> where the .ecc puts in enough redundant information to recover the
> underlying img.gz even when sectors or data are missing. If no such
> tool/format exists then two copies should be enough to recover all of an
> .img.gz so long as the same data wasn't lost on both media, and if bad
> DVD sectors always come back as "failed read", never ever showing up as
> a good read but actually containing bad data. Perhaps the frame
> checksum on a DVD is enough to guarantee that?
I use tar, gzip/bzip2, split - for creating a number of files of more or
less similar lenghts (like, 50 megs or 100 megs, but usually 50).
After that, I make par2 recovery files with par2cmdline tools (they make
use of Solomon-Reed error correction)
I am unable to find par2cmdline via google ATM, but they should be
And last but not least, I burn it all (data + pars). HTH.
** A C programmer asked whether computer had Buddha's nature. **
** As the answer, master did "rm -rif" on the programmer's home **
** directory. And then the C programmer became enlightened... **
** Tomasz Rola mailto:tomasz_rola at bigfoot.com **
More information about the Beowulf