[Beowulf] MS HPC... Oh dear...

[Jim Lux]

At 09:02 PM 6/11/2006, Joe Landman wrote:
>(sorry in advance for the length)
>
>Gerry Creager N5JXS wrote:
> > Ooh!  ISO-9002 Buzzword Compliant marketing.
>
>Hmmm.... not astroturfing here.  I opine on http://scalability.org/?p=69
>.  A few others have linked to this, so we are getting some traffic.
>
>Specifically, I am of the opinion that the sentence "but until now it
>has been too expensive and too difficult for many people to use
>effectively" is factually wrong.   The reasoning is very simple, and
>borne out by existing data.

Read it in a MS centric context...  It HAS been too expensive and difficult 
to use effectively, if you start with the premise that you're going to use 
only MS products.  Although, there has been a version of MPICH out there 
that runs on NT 4.0, etc. for years now, there are a number of other 
painful things when trying to build a MS Windows cluster, having mostly to 
do with cluster management (seeing what's going on on the other nodes, 
managing configurations). A lot of MS "clusters" are really more like 
"networks of workstations that happen to be running intercommunicating 
software" and are conceptually, little different than Seti at home. (well, 
maybe a bit more sophisticated..)



>If the statement of too hard could be applied to the market, one would
>need to ask exactly what people were buying that was not too hard which
>is generating all that growth.  Since we know the answer (linux
>clusters), they must not be too hard to use.  The systems we put
>together for our customers who don't care what is under the hood looks a
>great deal like a large windows disk (or disks) and a web page.  Those
>who care about the details prefer the command line.

I suspect that the entire cluster market is a tiny, tiny pimple on the 
behind of MS total sales, and the current cluster usage is a tiny fraction 
of that.  And of that total market, there's no question that some part of 
it does find clustering *easy enough*, hence the growth. However, for all 
you know, there might be 10 times as many people out who *think* (but don't 
necessarily *know*) that it's possible.

There are an enormous number of people who (for a variety of fairly good 
reasons) conceive of solving their problem using MS software tools (Visual 
C#, e.g.).  Having made a casual survey of some C# programmers I know, I 
discovered that the possibility of using a cluster to solve some of their 
problems wasn't even on the radar screen.


>All this said, and not to disagree with Doug Eadline and others on the
>technical details, I do think Microsoft has something to offer here, but

<snip>

>They do have a number of very hard hills to climb, specifically pricing
>compared to competitors, technological feature lists, interoperability,
>security, and stability.  Most of these are going to work against it.
>It would be unwise to count them out of the game though.  Anyone
>remember or still use Lotus 123?  Wordperfect?  May take them a while,
>and they are persistent.  With very deep pockets, lots of patience, and
>the ability to purchase talent.
>
>Linux was able to effectively kill Unix by presenting a single API to
>write to, a simple stack to deal with, a much larger potential installed
>base, a lower cost of acquisition.  Microsoft has learned from this.
>Assume that this is their direction.  The arguments they presented to me
>involved driving a wedge between various linux distros, and painting the
>Linux scene in a similar manner.  Their MPI argument (to many stacks)
>was not a good one, as the same problem exists on windows.  But the
>point is one that I and many others have complained about at some point
>in time or the other.  You have different MPI stacks which are binary
>incompatible.  Which means if the PathScale folks came out with a new
>hardware device to accelerate networking for folks like LSTC, then the
>LSTC folks have to relink their app against the new stack.  Which is
>exactly what happened.  While some folks here defend this, I want to
>note that end users don't give a rip about that.  They want the new
>fangled hardware to work.  Right away.  Without a rebuild of the app.
>So do the vendors.

Exactly.. Load up Visual Studio, Cluster Edition, load the app, compile and go.



>What Microsoft will do is to take away as much of this as they can.  I
>haven't seen it yet, but I believe they will offer MPICH as a DLL, so if
>PathScale wants to work along side some other device, you can select
>this at runtime, and just have it work.  This is a nice idea.

I'd guess more as a loadable component within the ".NET framework".

>A

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875