[Beowulf] Is there really a need for Exascale?

[Lux, Jim (337C)]

On 11/28/12 10:14 PM, "Mark Hahn" <hahn at mcmaster.ca> wrote:

>
>let me put it this way: when you buy a server from HP or Dell, you're
>getting a box optimized to maximize their business: give the customer
>warm fuzzies, minimize returns and service under warranty, etc.
>until recently, it would have integrated scsi, for instance, which
>tends not to be relevant to HPC.  heck, integrated video is not even
>strictly HPC-relevant.


Yes, but on a "buying FLOPS cheaply", you make the trade of low price due
to consumer commodity development vs non-HPC-optimized design

>
>>>  But I wonder what will drive further improvements in
>>> reducing power usage by several orders of magnitude.
>
>several orders of magnitude?  I'm not sure why this is even believed
>to be possible.  servers have improved from, say, >300W a few years
>ago to more like 250 today.  power-performance has improved a lot more,
>of course, but that's certainly not due to anything done at the MB level.

Actually, there are some MB design aspects.. Increased integration reduces
power consumption because a big cost is going off/on chip, so the more you
put on one die, the lower the power consumption.


>
>>> I've heard
>>> the suggestion that computers in cell phones will be the mass
>>> market that leads to low-power hardware suitable for supercomputers.
>
>doh.  sorta.  obviously mobile cares about power because of batteries.
>it's less clear that "wired" computers will ever care that much -
>how much will people notice their desktop taking 20 rather than 30W?
>
>>> But the cell phone components do not cover the same range as
>>> supercomputer components.
>>
>> But is that really true.   Sure, the processor in a cellphone is slower
>> than say a typical modern PC CPU.. But, given appropriate software, is
>>it
>> a better $/FLOPS or W/FLOPS deal to get 100 cellphone CPUs or 1
>>superduper
>> PC CPU?
>
>again, every one knows you get better performance/power if you go low.
>but too-slow processors are a problem to connect efficiently.


Light time bites you eventually when you go very low power. 1000 computers
that are 1cm across running at 1 Mhz will be beaten by 1 computer that is
5cm across running at 1 Ghz.





>
>>> What do others in the mailing list see as the trend?  Does the
>>> development of mass-market consumer products suffice for meeting
>>> the needs of the HPC community during this decade and the next?
>
>consumers don't buy IB, and it's the standard HPC interconnect.  beowulf
>has always been about repurposing commoditized parts where possible,
>but not exclusively.

I would say "beowulf has evolved".. The original Beowulfs were all about
commodity hardware, exclusively weren't they.  (leaving aside early
interesting stuff like PAPERS)

If you said "HPL Cluster Computing" I'd agree with your statement.


>
>> At some point, light speed becomes the limiting factor, and for that,
>> reducing physical size is important.
>
>we're quite a way away from that.  I don't see a lot of pressure to
>improve fabrics below 1 us latency (or so), ie, 1000 light-feet.

With conventional interconnects, it's a bit worse than that.  First off,
the distance isn't straight line (down from the chip to the mobo, across
the mobo to the connector, up to the connector, through the cable, through
the switch, through the cable...)

I'd say you're more in the 30 meter straight line distance range.  And
there's a fair number of clusters out there that are more than 30 meters
on a side.  And remember that this discussion is in the context of
exascale computing.  100,000 processors, even at 100 processors/19" rack,
is a square that is 30 racks on a side.  If you could densely pack them
that much, even that is in the 1 microsecond cable propagation ballpark.



>
>> Consumer gear is heading smaller, in
>> general (viz PC mobos getting smaller over the years),
>
>mainly due to integration, not anything else.  intel put cache onchip
>because it made performance sense, not because it freed up a few
>sq inches of motherboard.


I'd say both.  Consumer equipment is incredibly price sensitive, and
consumers like small things. There is HUGE pressure to get the tower PC of
yore (which was basically a PC/AT standing on its side) into something
that is the size of a Nintendo Wii.

Not only is the raw sheet metal cost less, but shipping costs less (how
many tower cases in a container vs how many Wiis).  And, of course, PC
board real estate is expensive.  Higher integration means that you can
have fewer onboard interconnects.  That might let you reduce the number of
layers, and it certainly reduces the number of square inches of PWB you're
buying.  Reducing layers greatly increases yield on the boards.


>
>> production density.  Consumer markets have the advantage of enormous
>> volumes to spread the very high non-recurring-engineering cost over.
>
>actually, the remarkable thing is that it seems to cost a lot less
>to produce custom parts these days.  if you (like facebook or google)
>want a custom motherboard, I bet there are 20 companies that will
>design and produce it for you for much less cost than 10 years ago...

Yes, to a point.  Of those 20 companies, 15 will give you a board that
isn't reliable under all temperatures or has weird timing issues.
They'll, maybe, be able to respin it a few times.

The actual production cost has come down a lot.  And the productivity of a
single designer with modern tools is much greater (ask one of those young
twerps standing on my lawn if they know what RubyLith is, or why it's
called tape-out)


>
>> delays).  How much is that worth?  10 million? 100 million?
>>(engineering
>> a new cellphone probably costs in the area of 10M, for context..  The
>
>I'm not so sure about that.  take the recent nexus4 phone: it's pretty
>much the same range of off-the-shelf components as Apple or Samsung use.
>the trick, as with any consumer product, is to find the right design
>to stimulate the purchase reflex.  for a long time, only Apple seemed
>to know how to do this consistently.

I think the internal design is much more complex and timeconsuming than
you think. It might take two or three spins of the board design til it
works consistently, for instance.
>