Beowulf in a Box

[Kragen]

On Tue, 29 Sep 1998, Mark Brinicombe wrote:
> On Mon, 28 Sep 1998, Kragen wrote:
> > On Mon, 28 Sep 1998, Bob Drzyzgula wrote:
> > > On Sun, Sep 27, 1998 at 08:15:59PM -0400, Kragen wrote:
> > 
> > I know nothing.  I assume a broadcast would work by first sending to
> > the first CPU, then the second, then the third . . .
> 
> Something like this. Alot depends on exactly how the meesaging is done but
> typically if the data is held in shared memory you just need to keep
> reference counts and send a message to each process in the network etc.

My understanding was that, at first, we were just going to do TCP/IP --
no shared memory.

There are cases where broadcasting a message would be faster than
keeping it in shared memory.  If you have ten megabytes of data that
each processor is going to want to read, it's quicker in theory to
broadcast it to all sixteen CPUs (assuming you have sixteen CPUs) with
a few messages totaling 10MB than to keep it in the host's memory and
have each slave CPU DMA it out of the host's memory separately, with
PCI traffic totaling about 160MB.

But I don't think PCI supports broadcast.

> > > Both PCI switches and this board are currently vapor,
> > > although it seems that an attempt to assemble these
> > > SA-110 boards may be soon 
> > 
> > Yes, I think it should be next month that the first CPU cards are
> > built, and the month after that that the main PCI boards go into
> > production.
> 
> Sorry if I am repeating stuff that has already been mentioned ;-)
> 
> What switches are being talked about ?

I think this part of the message was talking about PCI switches.
Fast Ethernet switches had been mentioned, too, because they *do* support
broadcast, and they also have better aggregate bandwidth if you have more
than ten CPUs or so.  (Or about six CPUs if you have two Fast Ethernet
channels per CPU.)

> I have one just joined the list so I missed the start of the thread.

Sorry -- the start of *that* thread was on a different mailing list.
I forwarded it here.

> A PCI switch is being looked at for the PCI card which will give each card
> 2 PCI busses which should help with communications as processors on the
> same card will be able to communicate with each other whilst the host PCI
> bus is being used for other communications etc.

If I understand correctly, it's really more of a PCI bridge -- it forwards
stuff from one PCI bus to another, but can only be connected to three
PCI buses, so it can't do the nifty Ethernet-switch thing where it's
forwarding numerous data streams at once.  Is that right?

The thing about switches (of the Fast Ethernet switch, or SP2 High-Speed
Switch variety) is that if you have forty machines connected to a
Fast Ethernet switch, then each machine can potentially be talking
at 100Mbps and listening at 100Mbps -- just as if each machine were
directly connected to the machine it wants to talk to, but with much
more flexibility.

> > > few new circuits of this complexity 
> > 
> > I think they are rather simple, actually; the CPU cards will have a
> > CPU, some memory chips, and a PCI bridge circuit on them, and (I'm told
> > thirdhand) that's it; the main PCI card will be even simpler.
> 
> The will also be some flash memory for boot firmware.

Sorry, I knew that, but didn't think of it.

Is that *really* all the circuitry that needs to go on there?
No auxiliary little chips, no stray diodes and capacitors?

> We don't want to go overboard with complexity to start with certainly you
> could arrange PCI busses in hypercubes etc. with lots of fancy switching
> but we are sticking to something fairly straight forward and something we
> can engineer on a short timescale.

Kibitzing without much knowledge, I humbly submit my opinion that that's
a damn smart thing to do.

> > The SODIMM-slotted Fast Ethernet adapters are also total vapor, though;
> > nobody's started trying to design them, although I can't imagine it'd
> > be hard -- they're just PCI Fast Ethernet cards with a funny form
> > factor.
> 
> This is just a suggestion of something that could be done and something
> that we have thought of. If there is a demand such things could appear.

Well, above you see the reasons why people might want it; it remains to
be seen how many people end up wanting some boards from the first run,
and how many people end up actually wanting Fast Ethernet daughtercards.

Do you know how many boards are "sold" now?

Kragen

-- 
<kragen@pobox.com>       Kragen Sitaker     <http://www.pobox.com/~kragen/>
I don't do .INI, .BAT, .DLL or .SYS files. I don't assign apps to files. I
don't configure peripherals or networks before using them. I have a computer
to do all that. I have a Macintosh, not a hobby. -- Fritz Anderson