[Beowulf] Rsync - checksums

[Ellis H. Wilson III]

On 6/18/19 11:00 AM, Bill Wichser wrote:
> Well thanks for THAT pointer!  Using --checksum-choice=none results in 
> speedup of somewhere between 2-3 times.  That's my validation of the 
> checksum theory things have been pointing towards.  Now to get xxhash 
> into rsync and I think we are all set.

Glad to hear that improved things for you.  I will caution against using 
a non-crypto hash for this though.  Unless you can verify xxhash offers 
at least as much protection against collisions as MD5, you may not be 
protecting your transfers sufficiently.  I haven't looked into xxhash 
enough to offer advice on this.

If you're already using rsync as just a piece of a larger framework, it 
really might be worth cooking up a script to md5sum the list in 
files-from on both sides in parallel to use all of your cores on both 
sides and then compare the results after the transfer.  Then you can 
just use --checksum-choice=none but still know your data made it safely 
to the other side.

Best,

ellis

-- 
Ellis H. Wilson III, Ph.D.
      www.ellisv3.com