[Beowulf] How Can Microsoft's HPC Server Succeed?

[Joe Landman]

Hi Anand

Anand Vaidya wrote:

[...]

> Having said that, I think that the Linux clustering scene needs a little
> competition, especially the for-fee ones. Apart from SDSC, not many
> innovations are happening.  I am not referring to standalone projects, where
> FOSS community has a lot of innovatio happening,  but rather one integrated
> Linux Cluster  on a DVD that gets you a cluster ready in 20mins, with no
> pain at all. ROCKS comes with its own problems, esp, wrt updates (which is
> why I stopped using ROCKS), however they are working on this one, AFAIK.

Hmmm.... I had a sense that no one is really willing to pay (in a 
traditional sense) for a cluster-on-a-DVD.  Well, maybe not this crowd ...

[...]

> So, here's what the FOSS community, especially,  vendors (RH, Novell) should
> be doing, specifically for a HPC oriented version:

Note that RH and Novell view HPC as a very tiny market, as most in this 
market exploit the FOSS variants, and don't pay for the OS install.  So 
getting their attention will be at least a little on the hard side.

That said, there are some folks doing these things now for their 
customers, and supporting them.

> 
> - remove all unwanted packaged (desktop software, multimedia, web browsers
> etc)

Check:  Most unwanted ... some folks want to rip mp3's in parallel on 
their cluster?  I dunno ...

> - package SGE, Ganglia,

Check:

> - a good clustering toolkit, maybe derived from ROCKS scripts (I am biased
> towards IBM xcat, because that is the only tool I use)

Check:  though define what "good" is.  xCat might not be everyone's 
impression of good.  Scyld does this as part of the distro itself.

> - LDAP as the default auth source, setup SSH for clusterwide passwordless
> logins by default

You will get arguments here on this.  LDAP can be needy.  Even when auto 
setup.  I like it, but some insist on NIS, or others.  By default, we 
would do a ldap_pam/NIS for internal, and hide this from the end user, 
unless they really wanted to see it for some reason.

> - easy integration to AD

Check:  :)

> - provide default environments , SGE queues etc) or easy to use scripts/gui

(big smile) Check:

> - Many users have low-complexity jobscripts. Provide a web UI (jetspeed2,
> tomcat ?). Let the advanced users tap the power of command line.

(big smile gets bigger) Check:

> - package a selection of top20 FLOSS science apps (Gromacs, Phylip, Blast,
> MPICH,  fasta, fftw etc)

Separate application bits from the usage bits ... packages should be 
easy to install and use in the cluster (gui/cli/...)

> - package and provide one click installation for restricted-ware such as
> NAMD, or commercial software such as Intel Compilers, Fluent, Amber etc. It
> CAN be done, Ubuntu has demonstrated how to do it well.

Understand that some of the non-commercial academic closed/restricted 
stuff could not be done via a one-click due to their particular 
licensing agreements.  That said, the one-click could be done after you 
get the tarball (yeah, I said tarball, not RPM).

> - package and provide easy install of a parallel filesystem such as GFS or
> Lustre

Noooo GFS.  Lustre is still too kernel dependent, though this is 
(slowly) changing.  You really don't want to go there yet.

Scyld will get you part of the way there.  Without being a commercial 
for them, others, us, you should see what you want, soon.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615