Mixed distros in one cluster

[Robert G. Brown]

On Sat, 3 Jun 2000, Simon Hogg wrote:

> Is there an inherent drawback in using different distributions in one 
> cluster (apart from more complicated maintenance)?
> 
> They should all work together, anyway, right?
> 
> Suse, Redhat and Debian is what I've got - are there any special 
> considerations for this combination that anyone can think of?

Better you than me, is all I can say;-).  Actually, I agree that it
might be fun to play with and compare the different distros in a
lab/beowulf setting -- if one had nothing else to do (like real work to
do ON the clusters), and I've suggested to some Intel Brass that they
consider funding such an effort at a public facility set up for that
very purpose.  However, I predict that you'll end up doing nearly three
times as much work solving the same problems three different ways and
building stuff for (possibly) three different library sets.  Actually,
RH and SuSE will probably coexist (both RPM based, similar libraries)
but I think that Debian and RH/SuSE will fight in various ways that will
require a lot of work, at least if you plan to make the software
offerings and user environment identical on all the platforms.

For truly large operations of any sort, heterogeneity is evil.  The more
that is different, the more that is nonstandard or custom, the more work
you have to do to provide a degree of homogeneity to benighted and
ignorant users.  I fought this fight for years with different Unices
(e.g. SunOS, Irix, AIX) in a single LAN and the distilled wisdom from
the experience is summarized as:

One person can do a pretty good job of installing, administering and
maintaining one operating system on one LAN.  If things are well set up
(that is, set up scalably with a fair degree of automation and
reasonably homogeneous hardware) the SIZE of the LAN can be pretty large
(hundreds of hosts) and one person can still manage the
hardware/software end of things.  However, user support doesn't scale so
well and a standalone systems person usually gets used up by users at
the expense of hardware before getting to that many hosts (unless a lot
of them are in a beowulf cluster so there are more machines than users).

One person CAN usually do two OS's (or two LANs in different
buildings/departments) but only if they do a less than perfect job on
one.  Too much to master, too much to duplicate, too much glue (or too
far to go and one place/group of people that suits you better).

One person can not generally do a good job with three.  Usually, having
three to keep running "acceptably" prevents one from having even one of
them running "excellently well".

Now with three Linuces you're not quite equivalent to three different
general Unices.  However, I'll bet that /etc is laid out differently,
that startup scripts are different, that different variables are set and
used, that they have different install tools, that different sets of
things are provided in a "default" installation and that different
packages are collected in different ways to support things like X,
gnome, WM's in general, news and mail tools, and possibly even compilers
and basic libraries.  It won't do to have one version of Gnome running
on RH and SuSE and a different one on Debian, or to have different
compiler revisions or kernels or module sets.  Just moving between
Slackware and Red Hat, I had to learn a huge amount and make fundamental
changes in the way I did various things.  Mostly for the better, I might
add, all though there are certainly still things that irritate me about
Red Hat.

> Of course, I will migrate everything to one distro at some time (probably 
> Debian) but different people want to 'play' with different distros, and 
> this is not a production cluster, so it might even make things more 
> interesting!

Remember the Chinese curse:  "May you live in interesting times";-) I
personally hope that your experience is interesting in only the best of
ways...

   rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu