Beowulf in a Box

[Mark Brinicombe]

On Mon, 28 Sep 1998, Kragen wrote:

> On Tue, 29 Sep 1998, Mark Brinicombe wrote:
> > On Mon, 28 Sep 1998, Kragen wrote:
> > 
> > Something like this. Alot depends on exactly how the meesaging is done but
> > typically if the data is held in shared memory you just need to keep
> > reference counts and send a message to each process in the network etc.
> 
> My understanding was that, at first, we were just going to do TCP/IP --
> no shared memory.
> 
> There are cases where broadcasting a message would be faster than
> keeping it in shared memory.  If you have ten megabytes of data that
> each processor is going to want to read, it's quicker in theory to
> broadcast it to all sixteen CPUs (assuming you have sixteen CPUs) with
> a few messages totaling 10MB than to keep it in the host's memory and
> have each slave CPU DMA it out of the host's memory separately, with
> PCI traffic totaling about 160MB.

I'm currently playing with several ideas on ways in which to do this. They
all have a number of performance issues. Typically shared memory will be
used for the messaging. Memory that is accessable to both the sending and
receiving processor can hold the message data. More specifically the
shared memory contains message lists so you then pass the messages between
processors using the inbound and outbound I2O FIFO registers. You also
have mailbox registers that could also be used for communications between
CPU cards.

> But I don't think PCI supports broadcast.

Nope. There are special cycles that are effectively broadcast cycles but
these have special uses and are not flexible enough to be of use.

> > > > Both PCI switches and this board are currently vapor,
> > > > although it seems that an attempt to assemble these
> > > > SA-110 boards may be soon 
> > > 
> > > Yes, I think it should be next month that the first CPU cards are
> > > built, and the month after that that the main PCI boards go into
> > > production.
> > 
> > Sorry if I am repeating stuff that has already been mentioned ;-)
> > 
> > What switches are being talked about ?
> 
> I think this part of the message was talking about PCI switches.
> Fast Ethernet switches had been mentioned, too, because they *do* support
> broadcast, and they also have better aggregate bandwidth if you have more
> than ten CPUs or so.  (Or about six CPUs if you have two Fast Ethernet
> channels per CPU.)
> 
> > I have one just joined the list so I missed the start of the thread.
> 
> Sorry -- the start of *that* thread was on a different mailing list.
> I forwarded it here.
> 
> > A PCI switch is being looked at for the PCI card which will give each card
> > 2 PCI busses which should help with communications as processors on the
> > same card will be able to communicate with each other whilst the host PCI
> > bus is being used for other communications etc.
> 
> If I understand correctly, it's really more of a PCI bridge -- it forwards
> stuff from one PCI bus to another, but can only be connected to three
> PCI buses, so it can't do the nifty Ethernet-switch thing where it's
> forwarding numerous data streams at once.  Is that right?

PCI switches are currently in their early days. The sort of switch I was
referring to is one where the secondary buses on the PCI card can
communicate between themselves privately whilst the primary bus is being
used by other devices. What this would allow is some communications to be
localised to the PCI card.

> > The will also be some flash memory for boot firmware.
> 
> Sorry, I knew that, but didn't think of it.
> 
> Is that *really* all the circuitry that needs to go on there?
> No auxiliary little chips, no stray diodes and capacitors?

Well there will be decoupling capacitors, a few resistors and possibly a
VCO and some sort of voltage regulation. I expect to be getting an update
from the hardware designer soon.

> Do you know how many boards are "sold" now?

Fraid not at the moment. 

Cheers,
				Mark