Computer Science research done on Beowulf class systems

[Walter B. Ligon III]

--------

The reason you hear more about scientific/engineering apps on parallel systems
is that they have been working on them for well over 30 years (jeez, I guess 
its
more like 40 years or more ...) so when one wants to implement parallel codes,
there is a lot of experience in doing that with large numeric codes.

There is a lot less experience in doing it with business apps, and much fewer
business app programmers that want to learn to program a parallel machine and
not too many companies are willing to take the risk of going with a parallel
system given they will have to port a lot of code and re-train programmers.

A lot less, but not none.  I know for example that back in the 80's some of
the big credit rating houses were working with parallel machines.  There are
also quite a few distributed systems out there.  Smaller companies are
sometimes willing to experiment with it in hopes of getting a leg up.

One big reason that businesses are reluctant (and the programmers who work
for them) is that it is not easy to quantify what (if anything) going parallel
will give back.  It isn't a situation of just plug in your parallel machine,
doing some porting, and BOOM! improved performance (or reliability or 
whatever).
You really have to know your system to know how and if a parallel 
implementation
will work for your app.  Also every problem can be parallelized in several
different ways.  So how do you know if you have done it the best way, or
even a decent way?

As an example, people are always posting to this group asking about using a
beowulf for a great-big web server.  Problem is, for most web servers it isn't
the computation that is the bottleneck - its the network.  A few really
large sites might need to spread the load a bit, but would probably be better
served my multiple servers that work together than a beowulf.  A beowulf
really doesn't have facilities to support really really large external network
traffic.  Of course, I am sure there ARE some web applications that COULD
make use of a beowulf.  For example, if my web page allowed users to browse
a large database of imagery and make image transformations - and finally
download the results then having a beowulf to do the file I/O and processing
might be a good idea.  Or, in the business arena, if my site does a lot if
complex database searches (more than just looking up a couple of records
in a table) then having those searches distributed by a beowulf on the
back side might make sense.

Most people in parallel processing research would like to see parallel
processing become more ubiquitous.  We tend to believe that eventually
all computers and all programs will be parallel.  A major step in doing
that is convincing our most conservative users this is a good thing, and
frankly we just aren't quite there yet.  Boy, it sure is a lot easier than
it was 10 years ago - but its still not the truly mainstream thing we'd like
it to be.

Anyway, those are my thoughts

Walt

> Brad, et. al., here is an idea/question from an applications guy without the
> technical know-how to answer it.
> 
> Everything on Beowulf seems to be related to large scientific/engineering
> applications.  Is Beowulf suitable for a more business oriented architecture
> of transaction processing against large relational databases?  How about a
> TCP/IP network connecting remote users to a Beowulf system with the database
> on a storage area network?
> 
> I have posed this question via fax and email to Red Hat and a professor
> whose web page seemed oriented toward more general problems.  I have had no
> responses so far, so either the topic is potentially so commercially
> lucrative that they don't want to talk about it, or so off the wall that it
> isn't worth a reply.
> 
> Thoughts, anyone?
> 

-- 
Dr. Walter B. Ligon III
Associate Professor
ECE Department
Clemson University