Neural network on beowulf - query

[Robert G. Brown]

On Fri, 23 Jun 2000, Ian McLeod wrote:

> Hi,
> 
> I am new to all of this, and I confess my lack of experience.
> 
> I am intrigued by the concept of running a neural network simulator over a
> beowulf cluster of a few friends PCs, but as I understand it from the manual
> and how-to's, this is easier said than done.
> 
> Only FORTRAN can be run in parralel if I read the literature correctly, C++
> will not work (or not well at all).
> 
> Is there any hope for parralel computing?

Cheeeze.  Of course there is hope for parallel computing, it's done all
the time by everybody on this list (or very nearly so).

First, I don't know what "literature" you are reading but just throw it
away if it says only FORTRAN will run in parallel.  Or use it to light
your fires next winter.  If this were an earlier, more rustic time I'd
suggest putting it in the cob box of the privy.

Second, since you have a LOT to learn before you can even ask sensible
questions, start reading.  The place to start with starting your reading
is probably the beowulf FAQ, followed by the beowulf HOWTO.  Both of
these cross reference numerous resources.  The main beowulf website
(www.beowulf.org) has additional useful links and resources.
http://www.phy.duke.edu/brahma provides both useful resources (including
a very incomplete but still useful draft book on beowulfery and several
talk/presentation/tutorial type things) as well as links to the FAQ, the
Howto, the beowulf site, the beowulf underground site, and other useful
resources.  Eventually you will likely want to buy and read "How to
Build a Beowulf" by Sterling, et. al. (MIT Press) and either or both of
"PVM" or "MPI" (also by MIT Press, don't remember the authors).

Whey you've waded through all that, come back and ask again to have
whatever you still don't understand explained.  Your guess is correct --
(the training of) neural network simulators can be run in parallel on a
beowulf cluster or more informal cluster consisting of you and your
friends' machines.  It can be programmed to do so in at least C (which
is what I use), C++ and Fortran, and with a bit more work or at a bit
lower speed one could program it to do so in perl, pascal, or pretty
much any programming environment with floating point and transcendental
support (need those nonlinear functions to morph the sum of the previous
layer's input into a neural output, and everything is in float) and ANY
kind of socket support.  Although it would be a bit insane to do so, one
could probably kludge something together in /bin/sh with awk for floats
and netpipes for the socket layer (yuk!).  In a good language and with a
fast network, one can even get gain (that is, the parallelized
application will run faster than a single-threaded, single hosted one).
This relies on your identifying parallelizable sections of the neural
training cycle (e.g. evaluating the error function on the training set)
and coding them to run in parallel.  This explicit example, by strange
chance, is the basis of one of the talks on brahma.  Which was written
in C, by the way...

Good luck.

    rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu