[Beowulf] copying big files

[Bernard Li]

I'd like to add a comment -- is the reason why this "issue" hasn't
been brought up as frequently as I think it should be mainly because a
lot of folks use distributed FS that eliminates the need to do one to
many file transfers?

Cheers,

Bernard

On Wed, Aug 13, 2008 at 4:14 PM, Bernard Li <bernard at vanhpc.org> wrote:
> Hi:
>
> On Tue, Aug 12, 2008 at 10:10 AM, Lombard, David N
> <dnlombar at ichips.intel.com> wrote:
>
>> See Brent Chen's pcp at <http://www.theether.org/work.html>
>> You'll want pcp, authd, and libe. Get gexec while you're at it...
>
> Dave!!  You beat me to mention pcp! :-)
>
> I really wished somebody would pick up the project and make it better.
>  I have tested it a few years back and it was really fast.  The
> general idea is that the host that has the original file will send to
> the neighbour and before that transfer completes the neighbour will
> send to its neighbour and so on and so forth.  But it has some
> shortcomings:
>
> 1) If one host in the path goes down, you need to start over again
> 2) The command option/interface is a little bit awkward, if I remember
> correctly you need to specify all the hosts manually in the command
> line
> etc.
>
> And yes, I agree BitTorrent is a simple solution that works well, that
> is why we integrated BitTorrent as part of the distribution mechanism
> of OS images in SystemImager.  However, it would be nice if someone
> actually wrote an application which eliminates the manual setup of the
> tracker, seed, etc. -- better yet, code something from scratch as
> BitTorrent cannot handle user/file permissions, and thus the way
> around it is to tar up the files you wanted transfer, and untar it
> after, which adds additional overhead.
>
> I look forward to trying out XGET, though.
>
> Cheers,
>
> Bernard
>