[Beowulf] Which distro for the cluster?
Many of your questions may have already been answered in earlier discussions or in the FAQ. The search results page will indicate current discussions as well as past list serves, articles, and papers.Robert G. Brown rgb at phy.duke.edu
Fri Dec 29 09:24:14 PST 2006
- Previous message: [Beowulf] Which distro for the cluster?
- Next message: [Beowulf] Which distro for the cluster?
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Fri, 29 Dec 2006, Geoff Jacobs wrote: > I'd rather have volatile user-level libraries and stable system level > software than vice versa. Centos users need to be introduced to the > lovely concept of backporting. The problem (one of many) is with operations like banks. In order for a bank to use a distro at all, it has to be audited for security at a horrendous level. If you change a single library, they have to audit the whole thing all over again. Costly and annoying, so RHEL "freezes" except for bugfixes because for companies like banks and other large operations, any change at all costs money. You can see the mentality running wild in lots of other places -- most "big iron" machine rooms were rife with it for a couple of decades, and even though I've been in this business one way or another for most of my professional life I >>still<< underestimate the length of time it will take for really beneficial changes to permeate the computing community. By years if not decades. I fully expected MS to be on the ropes at this point, being truly hammered by linux on all fronts, for example -- but linux keeps "missing" the mass desktop market by ever smaller increments even as it has finally produced systems that do pretty damn well on office desktops. I still view Linus's dream of world domination as a historical inevitability, mind you, I just no longer think that it will happen quite as catastrophically suddenly. Centos, of course, won't alter this pattern because diverging from RHEL also costs money and obviates the point for Centos users, who want the conservatism and update stream without the silly cost scaling or largely useless support. However, Centos "users" are largely sysadmins, not end users per se, and lots of them DO backport really important updates on an as needed basis. Fortunately, in many cases an FC6 src rpm will build just fine on a Centos 4 system, and rpmbuild --rebuild takes a few seconds to execute and drop the result into a yum-driven local "updates" repo. So I'd say most pro-grade shops already do this as needed. My problem with being conservative with a cluster distro is that it requires impeccable timing. If you happen to build your cluster right when the next release of RHEL happens to correspond with the next release of FC, it is auspicious. In that case both distros are up to date on the available kernel drivers and patches for your (presumably new and cutting edge) hardware, with the highest probability of a fortunate outcome. However, if you try to build a cluster with e.g. AMD64 nodes and the "wrong motherboard" on top of Centos/RHEL 4, all frozen and everything back when the motherboard and CPU itself really didn't exist, you have an excellent chance of discovering that they distro won't even complete an install, at least not with x86_64 binaries. Or it will install but its built in graphics adapter won't work. Or its sound card (which may not matter, but the point is clear). Then you've got a DOUBLE problem -- to use Centos you have to somehow backport regularly from a dynamically maintained kernel stream, or else avoid a potentially cost-efficient node architecture altogether, or else -- abandon Centos. The stars just aren't right for the conservative streams for something like the last year of each release if you are interested in running non-conservative hardware. The problem is REALLY evident for laptops -- there are really major changes in the way the kernel, rootspace, and userspace manages devices, changes that are absolutely necessary for us to be able to plug cameras, memory sticks, MP3 players, printers, bluetooth devices, and all of that right into the laptop and have it "just work". NetworkManager simply doesn't work for most laptops and wireless devices before FC5, and it doesn't really work "right" until you get to FC6 AND update to at least 0.6.4. On RHEL/Centos 4 (FC4 frozen, basically), well... One of the major disadvantages linux has had relative to WinXX over the years has been hardware support that lags, often by years, behind the WinXX standard. Because of the way linux is developed, the ONLY way one can fix this is to ride a horse that is rapidly co-developed as new hardware is released, and pray for ABI and API level standards in the hardware industry in front of your favorite brazen idol every night (something that is unlikely to work but might make you feel better:-). The fundamental "advantage" of FC6 is that its release timing actually matches up pretty well against the frenetic pace of new hardware development -- six to twelve month granularity means that you can "usually" by an off-the shelf laptop or computer and have a pretty good chance of it either being fully supported right away (if it is older than six months) or being fully supported within weeks to months -- maybe before you smash it with a sledgehammer out of sheer frustration. >From what I've seen, ubuntu/debian has a somewhat similar aspect, user driven to get that new hardware running even more aggressively than with FC (and with a lot of synergy, of course, even though the two communities in some respects resemble Sunnis vs the Shites in Iraq:-). SINCE they are user driven, they also tend to have lots of nifty userspace apps, and since we have entered the age of the massive, fully compatible, contributed package repo I expect FC7 to provide something on the order of 10K packages, maybe 70% of them square in userspace (and the rest libraries etc). This might even be the "nextgen" revolution -- Windows cannot yet provide fully transparent application installation (for money or not) over the network -- they have security issues, payment issues, installshield/automation issues, permission issues, and compatibility/library issues all to resolve before they get anywhere close to what yum and friends (or debian's older and also highly functional equivalents) can do already for linux. What the software companies that are stuck in the "RHEL grove" don't realize is that RPMs, yum and the idea of a repo enable them to set up a completely different software distribution paradigm, one that can in fact be built for and run on all the major RPM distros with minimal investment or risk on their part. Then don't "get it" yet. When they do, there could be an explosion in commercial grade, web-purchased linux software and something of a revolution in software distribution and maintenance (as this would obviously drive WinXX to clone/copy). Or not. Future cloudy, try again later. > Call me paranoid, but I don't like the idea of a Cadbury Cream Egg > security model (hard outer shell, soft gooey center). I won't say more, > 'cuz I feel like I've had this discussion before. Ooo, then you really don't like pretty much ANY of the traditional "true beowulf" designs. They are all pretty much cream eggs. Hell, lots of them use rsh without passwords, or open sockets with nothing like a serious handshaking layer to do things like distribute binary applications and data between nodes. Grid designs, of course, are another matter -- they tend to use e.g. ssh and so on but they have to because nodes are ultimately exposed to users, probably not in a chroot jail. Even so, has anyone really done a proper security audit of e.g. pvm or mpi? How difficult is it to take over a PVM virtual machine and insert your own binary? I suspect that it isn't that difficult, but I don't really know. Any comments, any experts out there? In the specific case of my house, anybody who gets to where they can actually bounce a packet off of my server is either inside its walls and hence has e.g. cracked e.g. WPA or my DSL firewall or one of my personal accounts elsewhere that hits the single (ssh) passthrough port. In all of these cases the battle is lost already, as I am God on my LAN of course, so a trivial password trap on my personal account would give them root everywhere in zero time. In fact, being a truly lazy individual who doesn't mind exposing his soft belly to the world, if they get root anywhere they've GOT it everywhere -- I have root set up to permit free ssh between all client/nodes so that I have to type a root password only once and can then run commands as root on any node from an xterms as one-liners. This security model is backed up by a threat of physical violence against my sons and their friends, who have carefully avoided learning linux at anything like the required level for cracking because they know I'd like them to, and the certain knowledge that my wife is doing very well if she can manage to crank up a web browser and read her mail without forgetting something and making me get up out of bed to help her at 5:30 am. So while I do appreciate your point on a production/professional network level, it really is irrelevant here. > Upgrade it, man. Once, when I was bored, I installed apt-rpm on a RH8 > machine to see what dist-upgrade looked like in the land of the Red Hat. > Interesting experience, and it worked just fine. There are three reasons I haven't upgraded it. One is sheer bandwidth. It takes three days or so to push FCX through my DSL link, and while I'm doing it all of my sons and wife and me myself scream because their ain't no bandwidth leftover for things like WoW and reading mail and working. This can be solved with a backpack disk and my laptop -- I can take my laptop into Duke and rsync mirror a primary mirror, current snapshot, with at worst a 100 Mbps network bottleneck (I actually think that the disk bottleneck might be slower, but it is still way faster than 384 kbps or thereabouts:-). The second is the bootstrapping problem. The system in question is my internal PXE/install server, a printer server, and an md raid fileserver. I really don't feel comfortable trying an RH9 -> FC6 "upgrade" in a single jump, and a clean reinstall requires that I preserve all the critical server information and restore it post upgrade. At the same time it would be truly lovely to rebuild the MD partitions from scratch, as I believe that MD has moved along a bit in the meantime. This is the third problem -- I need to construct a full backup of the /home partition, at least, which is around 100 GB and almost full. Hmmm, it might be nice to upgrade the RAID disks from 80 GB to 160's or 250's and get some breathing room at the same time, which requires a small capital investment -- say $300 or thereabouts. Fortunately I do have a SECOND backpack disk with 160 GB of capacity that I use as a backup, so I can do an rsync mirror to that of /home while I do the reinstall shuffle, with a bit of effort. All of this takes time, time, time. And I cannot begin to describe my life to you, but time is what I just don't got to spare unless my life depends on it. That's the level of triage here -- staunch the spurting arteries first and apply CPR as necessary -- the mere compound fractures and contusions have to wait. You might have noticed I've been strangely quiet on-list for the last six months or so... there is a reason:-) At the moment, evidently, I do have some time and am kind of catching up. Next week I might have even more time -- perhaps even the full day and change the upgrade will take. I actually do really want to do it -- both because I do want it to be nice and current and secure and because there are LOTS OF IMPROVEMENTS at the server level in the meantime -- managing e.g. printers with RH9 tools sucks for example, USB support is trans-dubious, md is iffy, and I'd like to be able to test out all sorts of things like the current version of samba, a radius server to be able to drop using PSK in WPA, and so on. So sure, I'll take your advice "any day now", but it isn't that simple a matter. >> within its supported year+, then just freeze it. Or freeze it until >> there is a strong REASON to upgrade it -- a miraculously improved >> libc, a new GSL that has routines and bugfixes you really need, >> superyum, bproc as a standard option, cernlib in extras (the latter a >> really good reason for at least SOME people to upgrade to FC6:-). > Or use a distro that backports security fixes into affected packages > while maintaining ABI and API stability. Gives you a frozen target for > your users and more peace of mind. No arguments. But remember, you say "users" because you're looking at topdown managed clusters with many users. There are lots of people with self-managed clusters with just a very few. And honestly, straightforward numerical code is generally cosmically portable -- I almost never even have to do a recompile to get it to work perfectly across upgrades. So YMMV as far as how important that stability is to users of any given cluster. There is a whole spectrum here, no simple or universal answers. >> Honestly, with a kickstart-based cluster, reinstalling a thousand >> nodes is a matter of preparing the (new) repo -- usually by rsync'ing >> one of the toplevel mirrors -- and debugging the old install on a >> single node until satisfied. One then has a choice between a yum >> upgrade or (I'd recommend instead) yum-distributing an "upgrade" >> package that sets up e.g. grub to do a new, clean, kickstart >> reinstall, and then triggers it. You could package the whole thing >> to go off automagically overnight and not even be present -- the next >> day you come in, your nodes are all upgraded. > Isn't automatic package management great. Like crack on gasoline. Truthfully, it is trans great. I started doing Unix admin in 1986, and have used just about every clumsy horrible scheme you can imagine to handle add-on open source packages without which Unix (of whatever vendor-supplied flavor) was pretty damn useless even way back then. They still don't have things QUITE as simple as they could be -- setting up a diskless boot network for pxe installs or standalone operation is still an expert-friendly sort of thing and not for the faint of heart or tyro -- but it is down to where a single relatively simple HOWTO or set of READMEs can guide a moderately talented sysadmin type through the process. With these tools, you can adminster at the theoretical/practical limit of scalability. One person can take care of literally hundreds of machines, either nodes or LAN clients, limited only by the need to provide USER support and by the rate of hardware failure. I could see a single person taking care of over a thousand nodes for a small and undemanding user community, with onsite service on all node hardware. I think Mark Hahn pushes this limit, as do various others on list. That's just awesome. If EVER corporate america twigs to the cost advantages of this sort of management scalability on TOP of free as in beer software for all standard needs in the office workplace... well, one day it will. Too much money involved for it not to. >> I used to include a "node install" in my standard dog and pony show >> for people come to visit our cluster -- I'd walk up to an idle node, >> reboot it into the PXE kickstart image, and talk about the fact that >> I was reinstalling it. We had a fast enough network and tight enough >> node image that usually the reinstall would finish about the same >> time that my spiel was finished. It was then immediately available >> for more work. Upgrades are just that easy. That's scalability. >> >> Warewulf makes it even easier -- build your new image, change a >> single pointer on the master/server, reboot the cluster. >> >> I wouldn't advise either running upgrades or freezes of FC for all >> cluster environments, but they certainly are reasonable alternatives >> for at least some. FC is far from laughable as a cluster distro. > What I'd like to see is an interested party which would implement a > good, long term security management program for FC(2n+b) releases. RH > obviously won't do this. I thought there was such a party, but I'm too lazy to google for it. I think Seth mentioned it on the yum or dulug list. It's the kind of thing a lot of people would pay for, actually. > Do _not_ start a contest like this with the Debian people. You _will_ lose. And I _won't_ care...;-) It took me two days to wade through extras in FC6, "shopping", and now there are another 500 packages I haven't even looked at a single time. The list of games on my laptop is something like three screenfuls long, and it would take me weeks to just explore the new applications I did install. And truthfully, the only reason I push FC is because (as noted above) it a) meets my needs pretty well; and b) has extremely scalable installation and maintenance; and c) (most important) I know how to install and manage it. I could probably manage debian as well, or mandriva, or SuSE, or Gentoo -- one advantage of being a 20 year administrator is I do know how everything works and where everything lives at the etc level beneath all GUI management tool gorp layers shovelled on top by a given distro -- but I'm lazy. Why learn YALD? One can be a master of one distro, or mediocre at several... > I haven't used a RH based machine which regularly synced against a > fast-moving package repository, so I can't really compare. :) Pretty much all of the current generation do this. Yum yum. Where one is welcome to argue about what constitutes a "fast-moving" repository. yum doesn't care, really. Everything else is up to the conservative versus experimental inclinations of the admin. > I personally believe more configuration is done on Debian systems in > package configuration than in the installer as compared with RH, but I > do agree with you mainly. It's way short of what FAI, replicator, and > system imager do too. The last time I looked at FAI with was Not Ready For Prime Time and languishing unloved. Of course this was a long time ago. I'm actually glad that it is loved. The same is true of replicators and system imagers -- I've written them myself (many years ago) and found them to be a royal PITA to maintain as things evolve, but at this point they SHOULD be pretty stable and functional. One day I'll play with them, as I'd really like to keep a standard network bootable image around to manage disk crashes on my personal systems, where I can't quite boot to get to a local disk to recover any data that might be still accessible. Yes there are lots of ways to do this and I do have several handy but a pure PXE boot target is very appealing. >> Yes, one can (re)invent many wheels to make all this happen -- >> package up stuff, rsync stuff, use cfengine (in FC6 extras:-), write >> bash or python scripts. Sheer torture. Been there, done that, long >> ago and never again. > Hey, some people like this. Some people compete in Japanese game shows. Yes, but from the point of view of perfect scaling theory, heterogeneity and nonstandard anything is all dark evil. Yes, many people like to lose themselves in customization hell, but there is a certain zen element here and Enlightment consists of realizing that all of this is Illusion and that there is a great Satori to be gained by following the right path.... OK, enough system admysticstration...;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
- Previous message: [Beowulf] Which distro for the cluster?
- Next message: [Beowulf] Which distro for the cluster?
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Beowulf mailing list