[Announce] The jobd Load Balancer

[Jakob Østergaard]

Hi all!

You may remember that I asked for a simple load balancing system for parallel
make jobs, about a week (or two) ago.

Some suggestions came up, and I was even offered the opportunity to beta-test a
commercial queuing system.  I decided to first play around with some of the
queuing systems already freely available out there, as well as various parallel
make variants.  I was certain that parallel makes was something a lot of people
did and therefore there would be mature and well functioning tools for the job
- as usual.

I need GNU Make, so BSD pmake is out.  Customs GNU Make doesn't work properly
and isn't maintained. PVM GNU Make may work, but didn't for me, I also have the
feeling that this is too much of a hack to be relied upon.  GNU Queue was
close, except that it breaks under load.  Generic NQS was too big (too slow for
short jobs, to complex).

Fixing GNU Queue wasn't an option for me, with all due respect that is by far
the ugliest code I've seen in a long time.

So I did what I originally wanted to avoid:  Wrote up a new load balancing
system from scratch.

It's very simple, providing an rsh like command ``jsh'' which instead of taking
a hostname argument (as rsh does) takes a job-type argument.  It communicates
with the jobd daemon running on the local host, and finds the best host for the
job-type given.  The job is then executed on this best host for the job.

For example, running the hostname command as a gcc-type job:
[joe at eagle joe]$ jsh -t gcc hostname
eagle
[joe at eagle joe]$ jsh -t gcc hostname
albatros

It's simple, efficient, and even somewhat secure. (I believe it is secure if
the network is physically secure and and nodes in the /etc/jobd.hosts file can
be trusted).

It is available at  http://ostenfeld.dk/~jakob/jobd/

The current version is 0.1, which should indicate that there is still work to
be done.  However, the system seems to work for me, and I'll be using it at
work the next few days to see how it fares.   There will be one major update
for the resource handling soon, but all in all I think the system is ready
for some use and feedback.  Hence this notice    :)

So if anyone besides me is sick and tired of waiting for those half-hour C++
compilations, here's a chance to justify a Beowulf for your boss   ;)

Cheers,
-- 
................................................................
: jakob at ostenfeld.dtu.dk  : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: