The FNN (Flat Neighborhood Network) paradox

[Patrick Geoffray]

Hi Miroslaw,

Miroslaw Gawenda wrote:

> The reason for this test is the potential problem when some nics access
> memory and PCI bus - I think so this can't give any reasonable speedup
> because 3 nics can't (probably) transmit and receive data in the same
> time.

If you use Fast Ethernet, 3 cards won't kill the PCI (10 MB/s per card,
full-duplex => 60 MB/s).
Nowadays, cheap 32 bits/33 MHz PCI can sustain 100-110 MB/s.

If you use IP, the host will die first, because of memory copies in the
IP stack / driver (wasting memory bus and cpu cycles) and interrupt
processing.
If you want to do message passing, I would recommand you some OS-bypass
communication layer like GAMMA
(http://www.disi.unige.it/project/gamma/).

> How many nics is ideal for clusters ?
One pipe per node. The size of the pipe depend of your needs (and your
budget).

Cheers.

-- 
Patrick Geoffray

---------------------------------------------------------------
|      Myricom Inc       |  University of Tennessee - CS Dept |
| 325 N Santa Anita Ave. |   Suite 203, 1122 Volunteer Blvd.  |
|   Arcadia, CA 91006    |      Knoxville, TN 37996-3450      |
|     (626) 821-5555     |      Tel/Fax : (865) 974-0482      |
---------------------------------------------------------------