disadvantages of a linux cluster

[Bob Drzyzgula]

On Wed, Nov 06, 2002 at 04:37:35PM -0500, Mark Hahn wrote:
> 
> from crunching a few of the numbers on CTC's website, it looks like 
> their per-node prices are in the >$50k range, compared to something 
> like $3-5K for the linux-whitebox approach.  to me, it looks like they
> simply front-end load their costs to make sustaining costs appear low.

I, too found the numbers in some of their press releases
interesting. For example, in this announcement:

  http://www.tc.cornell.edu/news/releases/2002/hps.asp,

we learn

  Cornell Theory Center (CTC) Receives $60
  Million In Resources Over Four Years to Deliver
  Compelling Performance for Compute-Intensive
  Commercial Applications

and

  CTC will double the size of its existing
  425-server Dell, Intel and Windows-based HPC
  clusters as a result of this agreement

and we have this quote:

  "This tremendous investment by Dell, Intel and
  Microsoft in the Cornell Theory Center is a
  true vote of confidence in the intellectual
  power of one of our State's finest academic
  institutions," said Governor Pataki.

I find all this mind-boggling. Simple math (and I
realize that the math must actually be far from simple)
would suggest that, in the absence of this grant from
the corporate sponsors, it would cost $60,000,000/425 =
$141,176 *per node* to build, program and operate this
cluster for four years. Perhaps some of this funding will
be used to operate the original nodes, or perhaps the
original nodes will be replaced and the grant is for 950
nodes. Even in that case, however, the per-node, four-year
cost is over $70,000.

I would be fascinated to know the details of CTC's
budget. How much of their operating expenses does this
$60,000,000 cover? Is Cornell (or the State of New York)
chipping in anything or is the entire bill being paid by
the corporate sponsors? How many salaries are being paid
out of this? Is there "cost recovery" income from the
projects doing their computing there, and if so how does
this figure into the budget? How much of the $60,000,000
figure is just the full list price of products being
provided for free? And especially, how much is the list
price of software that costs Microsoft nothing to provide?

I just can't begin to understand how this could possibly
be shown to be cost-competitive with a Linux cluster.
I know that, in most cluster installations, there are
typically large costs which remain hidden from view;
power, HVAC, per-square-foot office space, technical
staff which have been in place for years supporting other
systems, self-support and self-training by researchers,
pre-existing network infrastructures, etc. all are often
ignored when presenting to management the marginal cost
to implement a Linux cluster. While I'm ready to accept
that the $60,000,000 quoted in that press release includes
many of these things, the press release doesn't give us
any help in understanding how costs break down.

In any event, I suppose that if Dell, Intel and Microsoft
came to me and offered me $60,000,000 to put up and run a
Windows-based cluster for four years, I, too would have
to think real long and hard before I said no.  Hell,
there's always Cygwin.

--Bob