[Beowulf] fast file copying

[Geoff Galitz]

Hi folks,

During an HPC talk some years ago, I recall someone mentioned a tool  
which can copy large datasets across a cluster using a ring  
topology.  Perhaps someone here knows of this tool?

More to the point, we are pushing around datasets that are about  
1Gbyte.  The datasets are pushed out to dozens of nodes all at once  
and we foresee saturating the I/O system on our cluster as we grow.   
We are limited to using just the available disks and are looking for  
a reasonable solution that can support this kind of simultaneous  
access.   Currently we push the data out using rsync, but if I don't  
get any better ideas I may simply move to a pull system where the  
data is fetched by HTTP.  I can get better throttling that way, at  
least.

-geoff


Geoff Galitz
geoff at galitz.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.scyld.com/pipermail/beowulf/attachments/20070430/d6330b6f/attachment.html