batch systems with job deps (afterok)

[Jakob Østergaard]

On Tue, Sep 25, 2001 at 05:39:07PM +0200, hanzl at noel.feld.cvut.cz wrote:
> >> opensource batch system with support for job dependencies?
> >> ...
> >> OpenPBS (afterok feature of qsub) ... not easy to run on scyld
> >> ...
> >> All we need is to run 100 jobs in any order, then one job to merge
> >> results, then 100 jobs again and so on.
> >
> >If your jobs are relatively short lived (not days long), you could do this with
> >ANTS (http://unthought.net/antsd) and a Makefile to get the dependencies right.
> >
> >I think that could be a simple and efficient solution to your problem - but
> >note that ANTS is not meant to be a real batch queue system, and it therefore
> >does not have features such as queues, accounting, etc. etc.
> 
> Thanks, using 'make' this way really makes sense as our speech
> recognition training is something like huge build composed of
> five-minutes tasks.

antsd was made for tasks of that duration (C++ compilation in my case).

> 
> What I would miss with make (and is easy with 'afterok' job
> dependency) is the possibility of gradual creation of program to run -
> usually I am just a few steps ahead with program debugging - previous
> steps run while I prepare things which should follow (and once
> prepared, I would like them to run as soon as the previous step is
> finished).

If your runs take an input file and generate an output file from the input,
make should be able to see that it should not re-run already completed
jobs.

Thus, typing "make" again as your next task is ready should only result in
the non-completed tasks being run.

Or am I missing something ?

-- 
................................................................
:   jakob at unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: