[Beowulf] HD undetectable errors

[Nifty Tom Mitchell]

On Wed, Aug 26, 2009 at 12:18:17AM +0200, Reuti wrote:
> Am 25.08.2009 um 22:37 schrieb Nifty Tom Mitchell:
>
>> <snip>
>> However, It does make sense to me to generate and maintain site  
>> specific meta
>> data for all valuable data files to include both detection (yes tamper
>> detection too) and recovery codes.
>
> Sounds like using local par or par2 files along with their hash  
> information about the original files. Maybe this could be implemented as 
> FUSE filesystem for easy handling (which will automatically split the 
> files, create the hashes and any number of par files you like).
>
> -- Reuti

Interesting...  par2 looks close.
It may solve my worries with Cloud storage.

The differences between par and par2 are interesting.  The various
issues involving damaged files (even a single bit error) in the 
initial design of par were a limitation.

 http://www.par2.net/pardif.php

I can see that there has been a lot of work done already 
making it is a good place to start.  And yes slipping this
under a FUSE filesystem might hide the pain for local storage.


>> I would extend this to all data with
>> the hope that any problems might be seen first on inconsequential  
>> files.
>> Tripwire might be a good model for starting out on this.
>>
>> I should note that the three 'big' error rate problems I have worked on
>> in the past 25 years had their root cause in an issue not understood
>> or considered at design time so empirical data from the customer was
>> critical.  Data sheets and design document conclusions just missed the 
>> issue.
>> These experiences taught me to be cautious with storage statistics.
>>
>> Looming in the dark clouds is a need for owning your own data  
>> integrity.
>> It seems obvious to me in the growing business of cloud computing and 
>> cloud storage
>> that you need to "trust but verify" the integrity of your data.   My 
>> thought
>> on this is that external integrity methods are critical in the future.
>>
>> And do remember that "parity is for farmers."
>>
>>
>>
>> -- 
>> 	T o m  M i t c h e l l
>> 	Found me a new hat, now what?
>> a
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
>> Computing
>> To change your subscription (digest mode or unsubscribe) visit  
>> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
	T o m  M i t c h e l l 
	Found me a new hat, now what?