[Beowulf] Re: Finally, a solution for the 64 core 4TB RAM market

[Gerry Creager]

Mark Hahn wrote:
>>> I would guess that most people who currently have clusters would 
>>> rather get bigger/faster/cooler clusters, rather than go to SMP, 
>>> unless for some
>>> reason they have a fixed problem size.  possible, I guess.
>>
>> We intentionally built one cluster recently as a throughput system, 
>> with slow (ok, gigabit) interconnect, while the latest is "HPC" with 
>> DDR IB interconnect.
>>
>> We have throughput users (most jobs run on a single node, and can take 
>> advantage of the node's memory footprint).  A number of these are SMP 
>> or SMP-lite.  Did I mention computational chemistry?
> 
> sure - we have >3k users and lots of all these categories as well
> (and have also specialized our clusters).  but the point is that 
> 8-socket fat nodes are going to be more expensive; traditionally 
> nonlinearly more expensive.  current 4s boxes are more than 2x 2s cost.

I don't support 3k users (yet, things are heating up).  I see a 
particular need for some specialized cluster implementations, but I also 
see where a well-orchestrated mixed-use cluster of sufficient size can 
accommodate a lot of folks (Ranger comes to mind in a favorable light). 
  I don't see 8-socket nodes offering much real-world help to my task, 
but I can see them offering, perhaps, an interesting desktop cluster 
development environment for someone... or I could almost see a small 
proto-cluster of 'em for isolating development runs for users, who 
could, after successful testing, transfer working codes to a "real" 
cluster for normal use.  I do something similar on an old x86_64 cluster 
I've kept around against my boss' wishes.  It means I can develop 
without interfering with other users, occasionally crash the whole thing 
with impunity, etc.  More efficient if I had 4 h-node machines, than it 
is now.

> having fewer nodes is also a value, but mainly only if you wind up with
> single-digit numbers of nodes - if you're wrangling a cluster, it hardly
> matters whether it's 200 or 400 nodes.  (again, fat nodes have not 
> historically saved on power or space - at least not proportionally.)

Strongly agree.

>> We also have some folk interested in map-reduce, but I've not been 
>> able to accommodate them just yet.
> 
> yes, us too.  what are your thoughts on the kind of config that would suit
> them - just the google sort of layout?  (gigabit, I suppose.  probably 
> dual-socket, with however many 2G dimms will fit, and a couple large 
> local disks)

That's the direction I'm proceeding, but honestly, my real thoughts are 
"NO!" for use on a general purpose cluster.  I'm bringing up Hadoop On 
Demand, slowly, and I think it'll end up either as a rarely used app, or 
as a horrid waste of cluster resources.  I actually think it's a great 
app for a cluster of workstations, say, in a student lab after hours.

>> Depends on your mix of users.
> 
> I still think the market for 64c machines is relatively small.

-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843