[Beowulf] copying big files (Henning Fehrmann)

[Geoff Galitz]

I use dolly (http://www.cs.inf.ethz.ch/CoPs/patagonia/ and search for dolly)
from which nettee is forked and pdsh (http://sourceforge.net/projects/pdsh).
Both are great but dolly has certain advantages for my environment.  

In my case, I wrap it up in a service delivery tool for a 50 node cluster
where rapid deployment is key.  It is just some perl code that essentially
looks like this:

  foreach list_of_machines
    if they answer, add them to the list
  done
  login to the nodes via pdsh to start the dolly processes
  perform the transfer
  done!

There is actually a lot more to my code, but it is all environment specific
and deals with managing our custom apps.  I can push 1.6G (which includes an
svn checkout over the wire) to all nodes in approx 15 minutes.  The checkout
is the longest part, the actual file copy to the nodes is less than 5
minutes with our GigE network.  The nodes are busy processing while the data
transfer is in progress (this is an HA cluster).

When I researched this initially, I found there were actually a lot of
environment specific questions I needed answered, hence a lack of real
standardization on how these things are done.  Scaling is often the hardest
part... at least IMO.

I will say that my dream would be for something like dolly to get some sort
of transfer recovery mechanism, though I realize that would be quite
difficult in such a topology.  

As an aside, I know that the dolly author (Felix) reads this list.  I assume
dolly itself is now unmaintained?

Geoff Galitz
Blankenheim NRW, Deutschland
http://www.galitz.org


-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On
Behalf Of David Mathog
Sent: Freitag, 8. August 2008 18:12
To: beowulf at beowulf.org
Subject: [Beowulf] copying big files (Henning Fehrmann)

Henning Fehrmann <henning.fehrmann at aei.mpg.de> wrote:

> Coping a big file onto all nodes in a cluster is a rather
> common problem. I would have thought that there might be a
> standard tool for distributing the files in an efficient way. 
> So far, I haven't found one.