[Beowulf] Q: Experiences with high memory (64GB+) nodes?

[Brian Dobbins]

Hi everyone,

  I'm scouring through the web and reading the archives trying to track
down experiences and information from people with large-memory commodity
nodes.  A lab I know is looking at getting several 64GB or even 128GB
Opteron nodes, but unfortunately I've not been focused on hardware much
over the past few months, so I'm unsure as to what is out there and what
peoples experiences are.

  Has anyone recently purchased a 64GB+ Opteron (or Woodcrest) node
recently?  I'm curious about everything from the size of the DIMMs used
to motherboard advice to costs to any 'gotchas' that happened to arise.

  I imagine that the 128GB nodes would require 8-CPU boards (with 32 DIMM
slots) unless 8GB RAM modules are now available?  Presumably if they
are, they're prohibitively priced, too.

  Also with regards to motherboards, looking at the 4-CPU ones shows a
wide array of different capabilities for speeds of RAM.  I seem to
remember a discussion a while back (which I'll look up later) on the
effect of different speeds of DDR2 memory on performance, but what about
DDR2 (any speed) vs. DDR?  How much of a hit does performance take?

  I know, lots of questions, some of which I'll undoubtedly find answers
to in my own research, but the lab wants some sense of the information
by Wednesday, so I'm throwing this out to the collective wisdom of this
list.  ;)

  Thanks very much for any advice!

  Cheers,
  - Brian


Brian Dobbins
Yale Mechanical Engineering