[Beowulf] Rsync - checksums

[Ellis H. Wilson III]

On 6/17/19 1:35 PM, pellman.john at gmail.com wrote:
> I know that at one point, some Intel chips had instruction extensions 
> available to speed up SHA checksums by computing them directly in 
> hardware.  Might be worth looking into: 
> https://software.intel.com/en-us/articles/intel-sha-extensions

On my ancient desktop with a circa-2012 AMD processor md5sum executes on 
a cached 10GB file at roughly 650MB/s.  I would expect a reasonably 
recent CPU to crush this.  I expect one could max out a 25 or 40GbE link 
from a single client and a sufficiently fast CPU for m5sum on a modern 
processor.

I think we need more info from the OP describing why he thinks md5 is at 
fault before it makes sense to go down this rabbit hole.

But I agree with the general consensus that DR, especially for a 10PB 
system, is not best delivered via rsync alone.

Best,

ellis