Beowulf in a Box

[Simon Thorpe]

>On Sat, 26 Sep 1998, Kragen wrote:
>
>> On Sat, 26 Sep 1998, Kragen wrote:
>> > http://www.pobox.com/~kragen/sa-beowulf/
>>
>[snip]
>
>> I'm interested to hear other people's comments.  William Rankin
>> commented that such things have never gone anywhere, and I'm curious to
>> find out why.7
>
>Well, for one, there's a serious scalability problem.. Once you run out of
>PCI slots, you're finished. The nice thing about actual networking
>hardware is that you either get bigger a switch or add more switches
>connected via trunk lines with a small penalty in bandwidth and latency.

A couple of comments. First, there are PCI backplanes with anything up to
20 slots in them, so one isn't necessarilly limited by the number of slots
in a PC.

Second, I don't think of these boards as replacing the conventional
networking hardware. Nothing would stop you putting (say) four 8 processor
boards in a PC with a Gigabit network card and then connecting it up to
whatever else you've got handy.

Third. Perhaps you didn't notice that among the possible future
developments is the idea of having a fast ethernet SODIMM daughterboard.
That way, each 8 module boards could have 6 CPU modules and 2 ethernet
boards, one for each PCI bus. Will that reassure you?? I can see a lot of
point in using ethernet type messaging - not the least being the
possibility of broadcasting and multicasting messages. I for one will be
encouraging Chaltech to do this.

>I also think that without a native PCI-based MPI or other messaging layer,
>you're going to get killed by TCP/IP overhead. Gigabit ethernet (and
>myrinet) both only get around ~150 Mbits/sec TCP thoughput on a 200 Mhz
>Pentium Pro. (UDP is better, around 200Mbits/sec or so, but still only a
>5th of Gigabit ethernet wire speed).

Neil Carson has already said that one of the early things that they will
want to do is provide a shortcut for sending messages that cuts through all
those layers of protocol....

>There is also the problem of distributing computations to minimize
>communication latencies (if this is even possible for the problem you're
>dealing with). The aggregate bandwidth is great when transferring among
>procesors on the same card, but not when all 8 procs on a card need to
>talk to another board.

Without wanting to speculate too much on what Chaltech have in mind, I
think one can imagine implementations of hypercube type archictures based
on this sort of board in which typical latencies can be kept down in the
microsecond range...

>All of these problems could potentially be dealt with in software, but I
>suspect the development of this software will take on the order of 5 or 10
>years, even as an open source project. Some of the MOSIX load balancing
>algorithms might be quite usefull as a starting point. (I believe there
>are papers on most of them floating around)

Well - I do think that this is really unduly pessimisitic. The load
balancing problems are exactly the same ones that face anyone trying to use
Beowulf style clusters...

Cheers

Simon


__________________
Simon Thorpe
Centre de Recherche Cerveau et Cognition
133, route de Narbonne
31062 Toulouse France
Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09
http://www.cerco.ups-tlse.fr/private/simon.html
__________________