[Beowulf] Why I want a microsoft cluster...

[Jim Lux]

At 09:04 AM 11/26/2005, Mark Hahn wrote:
> > Ultimately though, to update the aphorism: "nobody ever got fired for
> > recommending MS"
>
>personally, I think the clustering world is a bit complacent, not unlike how
>unix was before Linux.  the terrarium is a barrier in both directions -
>because we are, for the most part structurally incompatible with WW,
>the competition is more tectonic, rather than teeth-at-neck.

Indeed... I agree.. the "current" user base for clusters is quite 
philosophically different from the "current" user base for MS products.



>so, getting back to the point, I don't really see from SC05, where WW will
>gain much from HPC clustering.  multiple vendors said that MSFT's goal was
>to insinuate itself into the low-end cluster market first, then move up.
>no real surprise there, but I have doubts about whether this makes sense.
>why?  because low-end clusters are mostly a mistake.
>
>at a university like mine, for instance, nearly _everyone_ realizes that it's
>insane for each researcher to buy/use/maintain his own little $50-500K
>cluster.  I see three clear reasons for this:
>         - the value of a cluster is some superlinear function of its size.
>         - the maintenance cost of a cluster is very sub-linear.
>         - most workloads are bursty.
>the first two factors encourage larger clusters; the latter means that bursts
>can be overlapped in a shared resource very nicely.


However, this sort of logic (the economies of scale) push clusters towards 
the model of the "mainframe supercomputer" in it's special machine room 
shrine tended by white garbed acolytes making sure the precious bodily 
fluids continue circulating.

One of the biggest values of the early clusters was that they let people 
"fool around" with supercomputing and get real work done, without hassling 
the instutional overheads.  Sure, they may have been non-optimized (in a 
FLOPS/dollar sense, certainly, and in others), but because they were 
*personal supercomputers*, nobody could complain.  They were invisible.

There is, I maintain, a real market for smallish clusters intended to be 
operated by and under the control of a single person.  In one sense, I'm 
wasting compute resources, but, I'm also doing this because my desktop CPU 
spends most of its time at <5% utilization.  Having that desktop under my 
personal control means that if something isn't working right, I can just 
abort it. Or, if I am seized by the desire to dump all the other stuff 
running temporarily and run that big FEM model, I can do that too.  No 
calling up the other users and asking if it's ok to kill their jobs. No 
justifying processor utilization to a section level committee, etc.

To recapitulate from previous discussions: a cluster appliance


>the other aspect of the MSFT-SC05 thing is that there doesn't appear to be
>any real synergy between the WW software ethos and clustering.  WW is all
>about plugging applications together - embedding activeX controls, etc.
>what would you plug into in a cluster?  there's nothing wrong with MSFT's
>plan to do a cluster environment based on SQL server, visual studio and MPI,
>but there's no real skeleton there, just connective tissue.  no Word/Excel

this is the real MS/Cluster disconnect... Clusters just don't seem to fit 
in the MS .NET world.  On the other hand, MS has a lot of smart people 
working there, and it's not out of the question that they'd figure out some 
way to leverage the idea of "commodity computers with commodity 
interconnects" in some MS useful way.

Jim