Request for pointers to books or articles on how to efficiently program Beowulf clusters

[Robert G. Brown]

On Tue, 8 May 2001, Ed Porter wrote:

> > What are some good books and good articles on how to write parallel
programs to run on Beowulf clusters (particularly relatively large
clusters) for a beginner at parallel programming.  My particular
interest relates to computations on large semantic networks.

The horse's mouth (so to speak;-) is Sterlin, Salmon, Becker and
Savarese, "How to build a beowulf cluster", MIT press.  In the same
MIT series are excellent primers on PVM and MPI.  Between them they
answer many if not all of your questions, although perhaps not
completely (you have a lot of questions!).

You can also look over some of the resources (talks, books, papers) on
or linked to the brahma site:

  http://www.phy.duke.edu/brahma

the most notable ones being the FAQ and HOWTO and the beowulf
underground site (which has links to still more resources).

Understand, though, that there are multiple answers to many of your
questions below, and whether or not you WANT to get very technical,
things like:

> -how threads are spawned and the costs in terms of time and
> computational resources of spawning a new instance of a thread,
>
> -how instances of the same thread communicate to each other or to
> instances of other threads (how do they know which machines other
> instance of the same thread may be on, and do they normally communicate
> through a message queue),
>
> -how does a process on one node read and write data into memory stored
> on another node,

etc. are VERY deep questions with very technical answers indeed, and
those answers vary depending on the kind of parallel programming library
or mechanism (e.g. PVM, MPI, DIPC, raw sockets) that is chosen.  This
applies to pretty much all of the questions below as well.

> -what control mechanisms are used for allocating computational and
> memory resources between competing processes,
>
> -how threads are terminated,
>
> -is there a mechanism for suspending a thread or computation on a task
> until data relative to it has been loaded into memory (for example if
> data from a list of different places has to be examined, can a prefetch
> be done for the data associated with each different location, and then
> can a list be kept of which of those different locations have their data
> in cache ready to be computed upon, so that CPU can keep busy processing
> the data that has already been loaded while data for other locations is
> being loaded into cache,
>
> -how large collections of data on hard disk are mapped into memory,
> and how the system knows which portions of the data are resident on the
> memories of which machines,
>
> -how 32 bit processors are used to address information on machines
> having a lot more than 2^32 bytes or words of RAM,
>
> -how to design parallel programs to run efficiently on Beowulf
> clusters,
>
> -how memory allocation and de-allocation is done,
>
> -how memory consistency is maintained when multiple processes on
> different machines are working on the same data, and
>
> -so on.

> So it would be particularly helpful if you could find some articles
> which don't require deep programming language or UNIX knowledge.

Yeah, doubt that I or anyone can help you there, although a lot of the
conceptual stuff doesn't require a deep technical knowledge.  However,
things like memory allocation and de-allocation and how memory
consistency is maintained require a pretty fair knowledge of cluster
computing to even know what the various acronym's you'll encounter stand
for.  You might also want to look at Greg Pfister's "In Search of
Clusters" book; it is a bit dated (what isn't that is older than 1 year
at this point?:-) but still a great book.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu