[Beowulf] fast file copying

[Joe Landman]

Geoff Galitz wrote:
> 
> Hi folks,
> 
> During an HPC talk some years ago, I recall someone mentioned a tool 
> which can copy large datasets across a cluster using a ring topology.  
> Perhaps someone here knows of this tool?

There are a few, commercial, and open source.

On the commercial side is exludus, xcp, and maybe one or two others. 
Exludus is basically a file pre-caching mechanism.  Java based.  xcp (by 
  Scalable) is MPI based.  It does a pretty good job of moving data.  \

On the open source side, I havent seen things other than the udp 
broadcast based tools (we had written one several years ago, named mcp), 
but anyone using a cluster will tell you that udp broadcast can be very 
detrimental to non-udp broadcast usage of the switch, say for logins, 
NFS, command and control, ...)

> 
> More to the point, we are pushing around datasets that are about 
> 1Gbyte.  The datasets are pushed out to dozens of nodes all at once and 
> we foresee saturating the I/O system on our cluster as we grow.  We are 
> limited to using just the available disks and are looking for a 
> reasonable solution that can support this kind of simultaneous access.  

xcp might help.

>  Currently we push the data out using rsync, but if I don't get any 
> better ideas I may simply move to a pull system where the data is 
> fetched by HTTP.  I can get better throttling that way, at least.

For a few dozen nodes, this might work.

Joe

> 
> -geoff
> 
> 
> Geoff Galitz
> geoff at galitz.org <mailto:geoff at galitz.org>
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452 or +1 866 888 3112
cell : +1 734 612 4615