[Beowulf] 512 nodes Myrinet cluster Challanges

[Mark Hahn]

> > in the cluster above, I choose a chassis (AIC) which has a large centrifugal
> 
> Which?  I noticed some of their designs redirect all heat from the powersupply
> into the side of the rack.
> 
> > blower, rather than a bunch of 40mm axial/muffin fans.  a much larger cluster
> > I'm working on now (768 nodes) has 14 40mm muffin fans in each node!  while
> > I know I can rely on the vendor (HP) to replace failures promptly and without
> > complaint, there's an interesting side-effect: power dissipation.  of 12 fans
> > pointing at the CPUs are actually paired inline, and each pair is rated to 
> > dissipate up to 20W.  so a node that idles at 210W and 265W under full load
> 
> Wow, 10 watts per 40mm fan?  Really?

the label on the fan gives V and A that work out to 20W per 
duplex 40x56mm fan.

what I've actually observed is that when the machine, under
constant load, goes from 6.4-6.7K to 9.3-9.9, its power 
consumption goes from a reasonable 281W to a worrisome 320W.

> I've a broshure from Sunon handy:
> 40x40x28mm  9200rpm 2.6 watts
> 40x40x28mm 11000rpm 4.0 watts
> 40x40x28mm 13000rpm 6.8 watts.

hah, how about 16800 rpm?  ;)

> The 56mm is 15.5 watts, but having 12 fans that large seems bordering

12 fans, but 6 duplex units, each up to 20W.  and yes, I agree
that 120W in fanage is insane.

> on insane.. I'd expect there wouldn't be much room for anything else.
> That would be a square of fans, er 16cm on a side

the machine has an arc of 6 of these duplex fans with a bit 
of a baffle controlling how the air gets to the CPUs.