[Beowulf] copying big files
bernard at vanhpc.org
Wed Aug 13 16:20:56 PDT 2008
I'd like to add a comment -- is the reason why this "issue" hasn't
been brought up as frequently as I think it should be mainly because a
lot of folks use distributed FS that eliminates the need to do one to
many file transfers?
On Wed, Aug 13, 2008 at 4:14 PM, Bernard Li <bernard at vanhpc.org> wrote:
> On Tue, Aug 12, 2008 at 10:10 AM, Lombard, David N
> <dnlombar at ichips.intel.com> wrote:
>> See Brent Chen's pcp at <http://www.theether.org/work.html>
>> You'll want pcp, authd, and libe. Get gexec while you're at it...
> Dave!! You beat me to mention pcp! :-)
> I really wished somebody would pick up the project and make it better.
> I have tested it a few years back and it was really fast. The
> general idea is that the host that has the original file will send to
> the neighbour and before that transfer completes the neighbour will
> send to its neighbour and so on and so forth. But it has some
> 1) If one host in the path goes down, you need to start over again
> 2) The command option/interface is a little bit awkward, if I remember
> correctly you need to specify all the hosts manually in the command
> And yes, I agree BitTorrent is a simple solution that works well, that
> is why we integrated BitTorrent as part of the distribution mechanism
> of OS images in SystemImager. However, it would be nice if someone
> actually wrote an application which eliminates the manual setup of the
> tracker, seed, etc. -- better yet, code something from scratch as
> BitTorrent cannot handle user/file permissions, and thus the way
> around it is to tar up the files you wanted transfer, and untar it
> after, which adds additional overhead.
> I look forward to trying out XGET, though.
More information about the Beowulf