Should SMP machine use 2 network card?

[Robert G. Brown]

On Thu, 17 Sep 1998, Jeong Hwan Park wrote:

> 
> I am plan to build 8 SMP machine beowulf cluster.
> total cpu number is 16.
> 
> If I use 1 nic card per smp node, does the network performance drop?
> 
> This is dilema when I have only 8 port fast ethernet switch but I want to
> build 16 processor beowulf cluster.
> 
> How can I solve this problem?

It depends.

"Network performance" per NIC doesn't drop; in fact it will generally
improve a bit because you have two CPU's to spread the network load
across.  Also, you get to use native IPC's between CPU's/nodes on each
dual box, which may more than compensate for sharing a NIC between two
CPU's to talk to other nodes.  However, the total bandwidth from a
single NIC per box is certainly shared, and if contention for the shared
resource gets to be too great you will start to suffer nonlinear losses
as one CPU blocks while the other uses the line (see previous message).

>From the previous message, it should be pretty clear that the proper
answer to your question involves some arithmetic on your part.  You need
to figure out the graining of your particular task, and derive a formula
that tells you whether or not your task will suffer from sharing a NIC
between two CPUs.  Fine grained tasks obviously won't.  Medium grain
tasks (where each node talks to only a one or two other nodes) probably
won't, or at least not so much that it matters.  Fine grain tasks almost
certainly will.

As to what to do about it:  First figure out if you have a problem,
something only you can do since only you know what kind of parallel task
you plan to run.  Measurements or experience beats the raw arithmetic --
if you post a detailed description of your task somebody can probably
tell you what kind of scaling to expect given 8x switched FE as a design
constraint.  If you don't have a problem, proceed to build it.

If you DO have a problem, there are lots of alternatives.  One is to buy
fewer single CPU nodes and a switch with more ports on it -- twelve
single PII nodes on a 12 port switch might well beat 8 duals on an 8
port switch for certain tasks.  Alternatively, you can stick (say) four
NICS in each node and crosswire them together in a hypernet of some sort
or another.  A 2d lattice of nodes, each connected to four "neighbors"
and wrapped onto a sphere, for example.  This solution is ideal if your
problem has a hypernet topology and doesn't need a switch at all.  Or
you can mix the two -- put eight duals on the switch and put additional
NICS in each box to hypernet behind the star.  Arrange your code so that
when any CPU on a box needs to talk to a host to which it is directly
connected, it uses the wire, and when it needs to talk to any host to
which it doesn't have a wire it uses the switch.  A pain in the ass to
optimize on the software side, but it could be done, if the relatively
trivial cost of a bigger FE switch is bigger than the cost of the
considerable amount of programming and engineering effort required to
otherwise beat the problem.

   rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb@phy.duke.edu