[Beowulf] Correct networking solution for 16-core nodes

[Tahir Malas]

Hi All,
With your previous suggestions 8 months ago we bought a Tyan S4881 server
with 8 dual-core Opteron CPUs with 64GB RAM. Now we will buy new ones (2
more for the time being) and we will eventually planning to form a cluster
from these servers, which will have at most 8 boxes. Now, as you guess, the
critical question that we seek for the answer is how to interconnect these
fat nodes for max efficiency? Our parallel program is heavily loaded with
many small-size and large-size P2P communications.  I feel that we need a
high performance network like Myrinet, IB, or Quadrics, not to waste 1us
latency and high bandwidth that we observe betw the 16 CPU cores. If anyone
setup such a cluster, can (s)he share his experience with us? In particular
I am curious about the answers for
*** How many NICs can or should we use per node? (SUN mbs now have 2 PCI-X
and 6 PCI-E slots, so we can theoretically insert 6 PCI-E NICs per node!)
*** Is the increase in the # of NICs necessarily increases the bandwith,
i.e. for example do we have 2 indep. comm links with 2 NICs?
Thanks in advance,
Tahir Malas
Bilkent University 
Electrical and Electronics Engineering Department