Beowulf - Single Board Computers? (long)

[Bob Drzyzgula]

> > OK, so this is something I know a little about; I've been
> > wanting to use SBCs for a Beowulf system for some time now,
> > and have been shopping the SBC market for a couple of years.
> 
> Well, I'm not sure if your restriction to "S"BC is killing you or
> what, but:

My "restriction" to single board computers is based on my
assesment of the likely space requirements for these units.
One major shortcoming of mainstream motherboards is the
requirement for (or at least assumption of) a dedicated
power supply per system. Between this and the mechanical
design of basic PCI cards, the best you can usually
do is a 2U chassis taking up the entire width of a 19"
machine rack.

Industrial (and perhaps military, I don't know) SBCs often
have much less complex power requriements. Many don't
even require 12V.  Look, for example, at the VSBC-6 from
Versalogic (http://www.versalogic.com/Ds/VSBC6.htm). Their
spec for running with a 300MHz K6 and 32MB of EDO memory
is 3.5A@5V, for 17.5W.  Using SDRAM probably requires
an additional 3.3V feed. What I was imagining was that
one could start with something like this and wrap it up
in a 6"x8.5"x1.5U metal box, with a couple of 5V fans
for airflow, and the only external connections being
15-pin D-sub (carrying one serial port, one Ethernet
port, and a relay control line for remote power-cycle)
and a four-pin Molex-type power connector (3.3V, 5V &
two grounds).  It may even be possible to include a 2.5"
EIDE disk drive and/or a PC-104 Plus Fast Ethernet module
inside each of these boxes. Put LILO into a little 2MB DOC,
and you would have no real need for a floppy. It should
be possible, I belive, to lay out as many as eight of
these on a rack-mount tray, giving eight CPUs per 2U of
rack height... perhaps 128 CPUs per 40U rack, allowing
room for network switches and power supplies. If properly
dressed out with sliding trays, it should be possible to
"hot swap" an entire compute node without disturbing the
rest of the Beowulf.

I don't see any way to achieve this kind of density with
mainstream ATX motherboards; The best I can figure to
do, given the alternatives I've seen, is around 40 CPUs
per rack, by using dual-processor ATX motherboards in 2U
boxen. At the same time, I do not expect that this
design would be cost-effective, unless the SBCs go down
in price or up in performance. I'd love to be proven
wrong, however.

> ...vendors out there selling systems which have a 90 degree "bus bender"
> that allows you to have 3.5 inch thick systems (www.aspsys.com), again
> using cheap commodity motherbaords.

DCG is one of these; I have several of their 2U chassis, with the
right-angle risers, and am quite happy with them. As I said in my
original message, I haven't yet found any good way to improve on this
solution today.

> The 3rd company I know of in the
> cluster market, Paralogic (www.xtreme-machines.com), seems to only
> sell a relatively compact conventional case with cards at...

That might be interesting; their website doesn't give much in the
way of inside-the-box pictures or diagrams, and they only list
commodity motherboards. Do you have more information from them?

> 2) Many _military_ vendors sell relatively small systems built on
> custom boards. They are very pricy but they are modern in their
> performance. Two example vendors of these are Alta and Mercury. And
> no, they don't have the limitation of "no communications or minimum 6
> boards"; they both are compact and still include 100 mbit and Myrinet
> options.
> 
> Perhaps the "industrial computer" segment is very different from the
> military segment. I don't see any of the military vendors in your url
> listing.

Correct. I am only familiar with the Industrial SBC market. Do you
have a list of URLs for Military vendors? I'd like to poke around
in those. 

> > * While all this ad-hoc stuff might work out well,
> > what would be unbelievably cool would be to write
> > a LAN-on-PCI implementation for Linux, that would
> > work with CompactPCI.
> 
> Why? I'd much rather have ethernet or Myrinet; switches always kill
> busses. For example:
> 
> > Since CompactPCI can
> > handle longer backplanes than card-edge PCI, you
> > could potentially get lots of CPUs in a single
> > rackmount enclosure, with a 133MBps network
> > backplane.
> 
> Which isn't that great; switched 100 mbit ethernet can quickly have a
> lot more total bandwidth than that. The actual performance of a
> lan-on-pci would probably be ~ 25% of peak, but we know we can get near
> 100% of peak out of switched ethernet. So the cross-over point is
> at... 3 cpus. And ethernet is cheap, cheap, cheap.

My figure was for 32-bit cPCI @ 33MHz which is what
seems generally avialable. But 64-bit PCI @ 66MHz should
be possible AFAIK, although perhaps not implemented
yet in cPCI. Also, PCI ring and star architectures are
under development, and switched PCI should be possible
(if bridges are possible, switches ought to be...)
LAN-on-PCI may not be the cat's meow for current PCIs,
but it seems to me that the future holds some promise,
especially since the PCI interface silicon is so cheap and
the volumes are so high -- I'd venture a guess that many
more PCI interfaces are implemented than even Ethernet
interfaces... my Mom's PC, for example, has at least
three PCI interfaces but no Ethernet interface.

This being said, I'm interested in where your 25%
figure comes from...  do you expect this to be a
limitation of the PCI architecture itself or simply a
limitation of bus architectures in general? What would
you say if the interconnect among PCI interfaces was a
dual counter-rotating token-passing ring rather than a
multiple-access bus? 

> > Not that any of this, AFAIK, is remotely
> > possible today.
> 
> CompactPCI-based systems are exactly what Alta sells into the military
> market. They are expensive enough that they have a separate product
> they sell to the cluster market. I'd be willing to bet that most of us
> are so cost-conscious that we won't want to pay for anything but a
> commodity board in a space-saving case, which seems to be pretty cheap
> today.

I know that cPCI is available, I mostly meant that
LAN-on-PCI wasn't. I agree backward and forward that
commodity boards are the only way to go right now from
a cost-effectiveness standpoint.

I would, however, like to see this change. They may be
cheap, but traditional motherboards take up a bunch of
space for stuff that is little or no use in a Beowulf;
having to run 110VAC to each processor is kind of a
drag, and makes some of the control stuff a bit more of
a hassle. If all you have to run into each box is maybe
40W of 5VDC power, simple microcontroller-run relays
on an RS-485 chain ought to do the trick; hell, you could
probably do it with a BASIC stamp and a 50 cent relay.

I can imagine that growth in the number of Beowulfs could
slow up a little once all the people who can afford
a conference room's worth of space already have one.
While I could see my office having a bunch of use
for a 200-node Beowulf, there simply is no place that
we could put one if we had to build them out of ATX
motherboards, and the HVAC guys would have a cow.

I see a lot of commonality of interest between
the Beowulf design and traditional embedded computer
designs. Also, greater interest in low-power, low-space but
high-performance computers for game playing and internet
access in homes could possibly result in some spin-offs
that would be useful in a Beowulf.

Thanks for the feedback.

--Bob

-- 
============================================================
Bob Drzyzgula                             It's not a problem
bob@drzyzgula.org                until something bad happens
============================================================