[Beowulf] Which distro for the cluster?

[Geoff Jacobs]

Forward:
I don't actually take any advocacy position on choice of distro. RH,
Debian, BSD, I don't care. Any contrary statements are made strictly in
the interest of the truth.

Robert G. Brown wrote:
> On Fri, 29 Dec 2006, Andrew M.A. Cater wrote:
<snip>
> 
> Also, how large are those speed advantages?  How many of them cannot
>  already be obtained by simply using a good commercial compiler and 
> spending some time tuning the application?  Very few tools (ATLAS 
> being a good example) really tune per microarchitecture.  The process
>  is not linear, and it is not easy.  Even ATLAS tunes "automatically"
>  more from a multidimensional gradient search based on certain 
> assumptions -- I don't think it would be easy to prove that the 
> optimum it reaches is a global optimum.
It most definitely isn't. Goto trounces it easily. ATLAS is the first
stab at an optimized BLAS library before the hand coders go to work.

> No, not joking at all.  FC is perfectly fine for a cluster, 
> especially one built with very new hardware (hardware likely to need 
> a very recent kernel and libraries to work at all) and actually 
> upgrades-by-one tend to work quite well at this point for systems 
> that haven't been overgooped with user-level crack or homemade stuff 
> overlaid outside of the RPM/repo/yum ritual.
> 
> Remember, a cluster node is likely to have a really, really boring 
> and very short package list.  We're not talking about major overhauls
>  in X or gnome or the almost five thousand packages in extras having 
> much impact -- it is more a matter of the kernel and basic libraries,
>  PVM and/or MPI and/or a few user's choice packages, maybe some 
> specialty libraries.  I'm guessing four or five very basic package 
> groups and a dozen individual packages and whatever dependencies they
>  pull in.  Or less.  The good thing about FC >>is<< the relatively 
> rapid renewal of at least some of the libraries -- one could die of 
> old age waiting for the latest version of the GSL, for example, to 
> get into RHEL/Centos.  So one possible strategy is to develop a very 
> conservative cluster image and upgrade every other FC release, which 
> is pretty much what Duke does with FC anyway.
I'd rather have volatile user-level libraries and stable system level
software than vice versa. Centos users need to be introduced to the
lovely concept of backporting.

> Also, plenty of folks on this list have done just fine running 
> "frozen" linux distros "as is" for years on cluster nodes.  If they 
> aren't broke, and live behind a firewall so security fixes aren't 
> terribly important, why fix them?  I've got a server upstairs (at 
> home) that is still running <blush> RH 9.  I keep meaning to upgrade 
> it, but I never have time to set up and safely solve the 
> bootstrapping problem involved, and it works fine (well inside a 
> firewall and physically secure).
Call me paranoid, but I don't like the idea of a Cadbury Cream Egg
security model (hard outer shell, soft gooey center). I won't say more,
'cuz I feel like I've had this discussion before.

Upgrade it, man. Once, when I was bored, I installed apt-rpm on a RH8
machine to see what dist-upgrade looked like in the land of the Red Hat.
Interesting experience, and it worked just fine.

> Similarly, I had nodes at Duke that ran RH 7.3 for something like 
> four years, until they were finally reinstalled with FC 2 or 
> thereabouts. Why not?  7.3 was stable and just plain "worked" on at 
> least these nodes; the nodes ran just fine without crashing and 
> supported near-continuous computation for that entire time.  So one 
> could also easily use FC-whatever by developing and fine tuning a 
> reasonably bulletproof cluster node configuration for YOUR hardware 
> within its supported year+, then just freeze it.  Or freeze it until 
> there is a strong REASON to upgrade it -- a miraculously improved 
> libc, a new GSL that has routines and bugfixes you really need, 
> superyum, bproc as a standard option, cernlib in extras (the latter a
>  really good reason for at least SOME people to upgrade to FC6:-).
Or use a distro that backports security fixes into affected packages
while maintaining ABI and API stability. Gives you a frozen target for
your users and more peace of mind.

> Honestly, with a kickstart-based cluster, reinstalling a thousand 
> nodes is a matter of preparing the (new) repo -- usually by rsync'ing
>  one of the toplevel mirrors -- and debugging the old install on a 
> single node until satisfied.  One then has a choice between a yum 
> upgrade or (I'd recommend instead) yum-distributing an "upgrade" 
> package that sets up e.g.  grub to do a new, clean, kickstart 
> reinstall, and then triggers it.  You could package the whole thing 
> to go off automagically overnight and not even be present -- the next
>  day you come in, your nodes are all upgraded.
Isn't automatic package management great. Like crack on gasoline.

> I used to include a "node install" in my standard dog and pony show 
> for people come to visit our cluster -- I'd walk up to an idle node, 
> reboot it into the PXE kickstart image, and talk about the fact that 
> I was reinstalling it.  We had a fast enough network and tight enough
>  node image that usually the reinstall would finish about the same 
> time that my spiel was finished.  It was then immediately available 
> for more work. Upgrades are just that easy.  That's scalability.
> 
> Warewulf makes it even easier -- build your new image, change a 
> single pointer on the master/server, reboot the cluster.
> 
> I wouldn't advise either running upgrades or freezes of FC for all 
> cluster environments, but they certainly are reasonable alternatives 
> for at least some.  FC is far from laughable as a cluster distro.
What I'd like to see is an interested party which would implement a
good, long term security management program for FC(2n+b) releases. RH
obviously won't do this.

> Yeah, I dunno about SuSE.  I tend to include it in any list because 
> it is a serious player and (as has been pointed out already in this 
> thread e.g. deleted below) only the serious players tend to attract 
> commercial/supported software companies.  Still, as long as it and RH
>  maintain ridiculously high prices (IMHO) for non-commercial 
> environments I have a hard time pushing either one native anywhere 
> but in a corporate environment or a non-commercial environment where 
> their line of support or a piece of software that "only" runs on e.g.
>  RHEL or SuSE is a critical issue.  Banks need super conservatism and
>  can afford to pay for it.  Cluster nodes can afford to be agile and 
> change, or not, as required by their function and environment, and 
> cluster builders in academe tend to be poor and highly cost senstive.
>  Most of them don't need to pay for either one.
<snip>
> Not to argue, but Scientific Linux is (like Centos) recompiled RHEL 
> and also has a large set of these tools including some 
> physics/astronomy related tools that were, at least, hard to find 
> other places.  However, FC 6 is pretty insane.  There are something 
> like 6500 packages total in the repo list I have selected in yumex on
>  my FC 6 laptop (FC itself, livna, extras, some Duke stuff, no 
> freshrpms.  This number seems to have increased by around 500 in the 
> last four weeks IIRC -- I'm guessing people keep adding stuff to 
> extras and maybe livna.  At this point FC 6 has e.g.  cernlib, 
> ganglia, and much more -- I'm guessing that anything that is in SL is
>  now in FC 6 extras as SL is too slow/conservative for a lot of
> people (as is the RHEL/Centos that is its base).
Do _not_ start a contest like this with the Debian people. You _will_ lose.

> Debian may well have more stuff, or better stuff for doing numerical
>  work -- I personally haven't done a detailed package-by-package 
> comparison and don't know.  I do know that only a tiny fraction of 
> all of the packages available in either one are likely to be relevant
>  to most cluster builders, and that it is VERY likely that anything 
> that is missing from either one can easily be packaged and added to 
> your "local" repo with far less work than what is involved in 
> learning a "new" distro if you're already used to one.
Agreed, and security is not as much of a concern with such user-level
programs, so these packages don't necessarily have to follow any
security patching regime.

> The bottom line is that I think that most people will find it easiest
>  to install the linux distro they are most used to and will find that
>  nearly any of them are adequate to the task, EXCEPT (as noted) 
> non-packaged or poorly packaged distros -- gentoo and slackware e.g. 
> Scaling is everything.  Scripted installs (ideally FAST scripted 
> installs) and fully automated maintenance from a common and 
> user-modifiable repo base are a necessity.  There is no question that
>  Debian has this.  There is also no question that most of the 
> RPM-based distros have it as well, and at this point with yum they 
> are pretty much AS easy to install and update and upgrade as Debian 
> ever has been.  So it ends up being a religious issue, not a 
> substantive one, except where economics or task specific 
> functionality kick in (which can necessitate a very specific distro 
> choice even if it is quite expensive).
I haven't used a RH based machine which regularly synced against a
fast-moving package repository, so I can't really compare. :)

<snip>
> Excellent advice.  Warewulf in particular will help you learn some of
>  the solutions that make a cluster scalable even if you opt for some
>  other paradigm in the end.
> 
> A "good" solution in all cases is one where you prototype with a 
> server and ONE node initially, and can install the other six or seven
>  by at most network booting them and going off to play with your wii 
> and drink a beer for a while.  Possibly a very short while.  If, of 
> course, you managed to nab a wii (we hypothesized that wii stands for
>  "where is it?" and not "wireless interactive interface" while 
> shopping before Christmas...;-).  And like beer.
Prototyping is absolutely necessary for any large-scale roll out. Better
to learn how to do it right.

> Yeah, kickstart is lovely.  It isn't quite perfect -- I personally 
> wish it were a two-phase install, with a short "uninterruptible" 
> installation of the basic package group and maybe X, followed by a 
> yum-based overlay installation of everything else that is entirely 
> interruptible and restartable.  But then, I <sigh> install over DSL 
> lines from home sometimes and get irritated if the install fails for 
> any reason before finishing, which over a full day of installation 
> isn't that unlikely...
> 
> Otherwise, though, it is quite decent.
> 
<snip>
> Oooo, that sounds a lot like using yum to do a RPM-based install from
>  a "naked" list of packages and PXE/diskless root.  Something that
> I'd do if my life depended on it, for sure, but way short of what 
> kickstart does and something likely to be a world of 
> fix-me-up-after-the-fact pain.  kickstart manages e.g. network 
> configuration, firewall setup, language setup, time setup, KVM setup 
> (or not), disk and raid setup (and properly layered mounting), 
> grup/boot setup, root account setup, more. The actual installation of
>  packages from a list is the easy part, at least at this point, given
>  dpkg and/or yum.
I personally believe more configuration is done on Debian systems in
package configuration than in the installer as compared with RH, but I
do agree with you mainly. It's way short of what FAI, replicator, and
system imager do too.

> Yes, one can (re)invent many wheels to make all this happen -- 
> package up stuff, rsync stuff, use cfengine (in FC6 extras:-), write 
> bash or python scripts.  Sheer torture.  Been there, done that, long 
> ago and never again.
Hey, some people like this. Some people compete in Japanese game shows.

> rgb

-- 
Geoffrey D. Jacobs