data transfer and condor

[Steven Berukoff]

Hi all,

We're looking at using Condor on our cluster for its checkpointing and
job-handling abilities, as the routines we're running don't require much
in the way of internode communication.  We have an NFS file server which
contains our entire fileset (something on the order of 100s of GB), a
master node for the cluster, and several nodes.  Outside of Condor, our
algorithm requires that each of the nodes get some subset of the data (on
the order of perhaps 100MB) and runs the analysis code on this data
segment.  Obviously, each node must gather its share of the data from the
NFS file server; this of course requires a large amount of network
traffic.  

Does anyone have a clever idea about scheduling the data
transfers so that it is accomplished in a reasonable fashion?  We were
hoping Condor provides this functionality to some degree, but it doesn't
seem to.

Thanks
Steve



=====
Steve Berukoff					tel: 49-331-5677233
Albert-Einstein-Institute			fax: 49-331-5677298
Am Muehlenberg 1, D14477 Golm, Germany		email:steveb at aei.mpg.de