Beowulf - Single Board Computers? (long)
Bob Drzyzgula
bob@drzyzgula.org
Sun, 20 Sep 1998 11:33:32 -0400
OK, so this is something I know a little about; I've been
wanting to use SBCs for a Beowulf system for some time now,
and have been shopping the SBC market for a couple of years.
I haven't quite got there yet, we're just now prototyping
the code on a simple PVM array of conventional systems, but
we have serious space considerations, and I'd like to go
the SBC route if possible. Sadly, I have yet to convince
myself that it is possible, or at least practical.
In my shopping, I've found the following;
* SBCs are expensive. Very expensive, compared to
mainstream ATX stuff. Most SBCs are intended as
"Industrial Computers", and industrial computing is
a very different animal than Doom-on-the-desktop.
Not that you *can't* use an industrial computer (IC)
for Doom-on-the-desktop, it is just that ICs have a
number of specific advantages that are a total waste
on the desktop or in a Beowulf array. To wit:
- ICs typically are designed to handle harsh
environmental conditions. Like temperature ranges
from 0-120C, near-constant vibration, intense
EMI baths, etc. Put a PC Chips motherboard in this
environment and it will probably disintegrate
in minutes.
- ICs are usually intended for embedded uses,
which usually implies long development and deployment
cycles. Among other things, this often results in
SBCs being quite far behind in technology. The 430HX
is a hot new product in these circles, for example. I
haven't yet seen a 440LX or VP-3 SBC, much less a
MVP3 or 450NX. (I'm not saying they aren't there,
just that I haven't yet seen them)
- As a consequence of the previous item, prices
follow a much shallower curve than they do in the
mainstream hardware market. 80486 boards are just
now falling into the sub-$250 range, while 186, V25,
and 386 boards are still viable, profitable products.
* Networking for ICs is more often than not just a
connectivity thing, and high-end performance is not
often an issue. On-board Ethernet chips are commonly of
the Realtek or Crystal variety. You generally have to
install your SBC into a backplane in order to connect
to a high-performance network card.
* The space savings can be somewhat illusory, especially
at the high end. Those full-size ISA style cards are
actually a special "PICMG" (PCI Industrial Computer
Manufactures Group) format cards. Very few (if any)
of them will run without being installed in a passive
backplane, and full-blown PICMG backplanes rarely are
less than six slots wide. You can often get dual-CPU
passive backplanes that will allow you to build two
complete computers in a single chassis, but the the
space you save won't be that much different than what
you get out of conventional hardware in a 2U chassis
such as that from http://www.dcginc.com. Four-CPU
backplanes are in fact available, but power distribution
becomes a major issue; it is very unlikely that the
power planes in the backplane PCB will be able to feed
four dual processor SBCs, and you will most likely
have to go to an external rack-mount power supply.
I say this based on an extensive conversation with an
engineer at one of the bigger IC companies that sells
quad-SBC backplanes.
* A new format called "PISA" (PCI/ISA) is starting to
show up, wherein the ISA and PCI lines are interleaved
in a single card edge, so that you can deliver both
buses into a single riser, and half-length SBCs can
be used. But again, you generally need a backplane to
run these.
* The best bet from the standpoint of building a tiny
Beowulf node is, in my mind, a half-size SBC without a
PICMG or PISA connector. This is severely limiting in
terms of I/O, but, as described above, adding high-end
I/O to SBCs runs into high space and cash utilization
issues. A half-size Pentium board (I have never seen
a P6 architecture board in this format) with built-in
Ethernet and EIDE could probably be integrated into a
cabinet the size of a child's shoe-box, since most of
them can be run without a backplane. Most such devices
will also include a bootable Flash disk, so you could
have a lilo-enabled stub in there that pulled in the OS
from NFS. As a bonus, some of these boards (the Ampro,
for example) support totally VGA-less operation;
you can reboot and tweak the CMOS settings on the
Ampro Little Board remotely through a serial line.
Not like you don't pay for this, however.
* Other extremely attractive technologies,
from a Beowulf perspective, are PC-104 and PC-104
Plus. PC-104 is a re-implementation of the ISA bus onto
a (approx) 4" square board, with the bus implemented
on stacking pin-and-socket connectors, with liberal
addition of ground pins to stabilize the bus in harsh
conditions. The 104 comes from the number of pins on the
bus. PC-104 Plus is a recent enhancement to the format,
which provides for a corresponding pin-and-socket
PCI bus on the other side of the card. Most PC-104
CPU cards use double-sided, 6-layer PCBs, and even at
that there ain't a whole hell of a lot of real estate,
so Pentium PC-104 boards are few and far between. Many
PICMG and half-ISA SBCs include PC-104 or PC-104 Plus
connectors, however, so the most common use of PC-104
is to provide expanded I/O to an SBC without having to
implement a backplane.
* As a little fantasy, the cards made by ZF Microsystems
and CellComputing would make really cool Beowulf
nodes if they were (a) fast enough (b) cheap enough,
and (c) had decent I/O bandwidth. I don't think any
of these criteria hold, however.
* Some of the big guys also are into the SBC game.
IBM, Digi^H^H^H^HCompaq, and especially Motorola
sell SBCs. Motorola sells some cool PowerPC SBCs that
might make nice Beowulf nodes, except that they are
very pricey, and I expect that getting the OS, drivers
and Beowulf enhancements stable for these boards would
take a little work.
* While all this ad-hoc stuff might work out well,
what would be unbelievably cool would be to write
a LAN-on-PCI implementation for Linux, that would
work with CompactPCI. (Or possibly LAN-on-VME, but
VME boards remain very, very expensive, and the
high-bandwidth VME implementations are even more
so. CompactPCI at least stands a chance of making
it into the moderate price range). I know that
LAN-on-VME implementations exist for other operating
systems, and I have heard of engineers hacking
on LAN-on-PCI technology. Since CompactPCI can
handle longer backplanes than card-edge PCI, you
could potentially get lots of CPUs in a single
rackmount enclosure, with a 133MBps network
backplane. Not that any of this, AFAIK, is remotely
possible today.
To me, all this comes down to "How many high-performance
CPUs can I get in a single, 40U machine rack, without
having it cost a fortune?" So far, I haven't found
anything that would work out better than a 2U ATX chassis
(e.g. http://www.dcginc.com/35rack.htm) populated with
dual-processor Pentium II motherboards. Still, I can't
believe that there isn't a better solution. It's just
that every time I look, I can't find the perfect compute
engine that will do it.
All this having been said, here's a few URLs to
tickle your fancy. This is a very truncated
list compared to what is actually out there.
http://www.eg3.com is one of the very best
places to go if you want to find more.
[Note, I haven't reviewed all the sites I list
below for a few months. If anyone does so and
finds something new and cool, please let me know]
Consortiums & trade press:
----------------------------
http://www.picmg.org/
http://www.pc104.org/
http://www.embedded.com/
http://www.controlled.com/
http://www.eg3.com/
Manufacturers (*):
----------------------------
http://www.advantech-usa.com/
http://www.trentontech.com/
http://www.octagonsystems.com/
http://www.ampro.com/
http://www.jump.de/
http://www.portwell.com.tw/
http://www.ziatech.com/
http://www.winsystems.com/
http://www.adastra.com/
http://www.teknor.com/
http://www.adlogic-pc104.com/
http://www.inside-us.com/
http://www.versalogic.com/
http://www.parvus.com/
http://www.zfmicro.com/
http://www.cellcomputing.com/
http://www.mcg.mot.com/WebOS/omf/GSS/MCG/index.html
Vendors (*):
----------------------------
http://www.emacinc.com/ (online pricing!)
http://www.technoland.com/
http://www.appro.com/
http://www.siliconrax.com/
http://www.emj.com/
http://www.marshall.com/ (online pricing for Octagon boards)
http://www.indcompsrc.com/
(*) This is my best guess. I could have some manufactures
listed as vendors and vice versa. These guys don't tend to
go out of their way to make the distinction clear.
--Bob
--
============================================================
Bob Drzyzgula It's not a problem
bob@drzyzgula.org until something bad happens
============================================================