[Beowulf] RE: Capitalization Rates - How often should you replace a cluster? (resent - 1st sending wasn't posted ).

[Greg Lindahl]

On Wed, Jan 21, 2009 at 10:57:42AM -0500, Mark Hahn wrote:

> I guess our premises are quite different.

Yes, they are. Hence your statement about "obviously burning money"
was a bit of an over-generalization ;)

There are HPC systems that want UPSes, namely ones that do
mission-critical things, like weather forecasts, and spooky work.
Most university HPC systems don't fall into that category. Most
business websites do fall into the UPS category. But other than that,
my cluster looks pretty much like an HPC cluster, albeit one with
only gbit ethernet and lots of local disk.

> first, I don't see why bandwidth
> would be a huge issue for HPC.

The major US academic supercomputing centers are usually a majority of
the initial nodes of each new, faster Internet incarnation (currently
mostly 40 gbit links).  You guys are unusual.

BTW I didn't include bandwidth cost in the rent: I pay $7,000/month
per 1 gbit link. However, it is important to be in a building that
already has fiber drops from multiple providers to get that kind of
rate.

> another of my premises is that UPS is generally unnecessary, at least  
> for compute nodes.  I have no idea how it would affect pricing at a colo,
> if you said "I'd like 46U/rack with line power and 2U with UPS."

In my colo, the whole thing has the same services. Since I only have
100s of systems, and that's a tiny % of the colo, there's no
cost-effective way to have a custom setup beyond the # of power
circuits per rack.

To answer Joe's question, I think 365 Main (the one with the big
outage) has been full for a while. I'm in a newish colo in Sunnyvale,
an ex-Global Crossing building. From what my friends tell me, I'm
paying the market rate.

-- greg