The FNN (Flat Neighborhood Network) paradox

[Jim Lux]

>
>Remember, even for a tetrahedral four node beowulf, you must spend
>perhaps $20 (or more, of course) per NIC, or $60 per node.  Four nodes
>require 12 NICs and six cables for a total cost of around $270 or more.


<nitpick>
A tetrahedron isn't a hypercube, is it?  A 4 node hypercube would have 2
links per element. 8nodes would require 3 links, etc. (i.e. log2(N) links
for N nodes), or, the number of links = number of dimensions.

I'm not sure what a higher dimension version of the tetrahedron would be
(having purged all that pure math stuff from my brain 20 years ago).
log2(N)+1 links?
</nitpick>

>
>A five port switch costs perhaps $70 (or less if you shop hard) -- eight
>port switches are as little as $80.  A switched port costs LESS than a
>NIC these days.  Admittedly these switches are likely to be
>store-and-forward with mediocre latency, but even better switches aren't
>that expensive anymore.  Add in only FOUR NICs and cables @$25 each, and
>you can get effortless connections for only $170 and have an extra port
>to connect up a head node or to another switch.

What are cheap/inexpensive switches that don't use store and forward?



>Channel bonding, on the other hand, solves a very different problem --
>how to get more raw internode bandwidth using any given kind of NIC.
>This is also "expensive" in human time and possibly system time, but it
>may be the cheapest (or only!) way to get internode bandwidth in certain
>exotic ranges.  If you have a parallel application that is not
>particularly sensitive to latency but needs huge interprocessor
>bandwidth to scale well, it can easily be your ticket to a functional
>design (for a largish but still COTS price).  If I recall what Don
>Becker once told me, your aggregate bandwidth increases, not quite
>linearly, for up to three NICs but the fourth (at the time of the
>discussion, not necessarily now) was either not worth it or actually
>decreased aggregate bandwidth a bit.

This discussion brings up an interesting question:


Say you had 2 8 port switches and wanted to interconnect, say, 8<=N<=4
processors with a pair of NICs in each.  What is the optimum arrangement for
channel bonding?  All processors send one NIC to one switch and the other to
the other?, or some sort of hybrid where you send both NICs to the same
switch. (obviously, one can scale this problem up... in today's market, the
correct answer would be to just buy a 16 port switch).  How well does
channel bonding deal with the potentially different latency through
different paths?  Does the various protocols deal well with messages
arriving out of sequence? Obviously, some layer of the protocol could be
responsible for buffering and rearranging, but that might not be the best
approach for some applications.