myrinet vs gigabit

[Patrick Geoffray]

Hi,

Woo Chat Ming wrote:

>  I am going to set up a 100-nodes beowulf cluster
> to do scientific simulation. Is Gigabit ethernet or
> Myrinet better ? Does anyone has performance comparation
> of them ?

Nate Fuhriman wrote:
 
> For some information on the speeds achievable with myrinet see
> 
> http://www.linuxnetworx.com/products/myrinet.html

This information is quite outdated. The link speed is 2 Gb/s + 2 Gb/s.
The performance with the GM interface is, on the product available now,
8 us and 250 MB/s application to application. MPI is 10 us and 220 MB/s,
TCP is 1 Gb/s and UDP 1.6 Gb/s with Linux 2.4.x.
 
The main differences are price and performance. 
The cost of a 100 nodes cluster with Myrinet is about ($1200 x 100) +
$50000 = $170000. 100 nodes fit in one switch with full bissection. The
same configuration with Gigabit Ethernet is much more expensive (with
some research on the web, I have found $300000 with a not-so-bad
bissection from ExtremeNetworks). Usually, the price cross-over point
between GigE and Myrinet is around 32 nodes.
The protocol layer used in production mode with GigE is IP. That means
higher latency, lower bandwidth and high CPU load (I hope the 0-copy IP
stack that Linux folks are testing would change that). Myrinet provides
IP, but MPI over GM (OS by-pass) is certainly more attracting for
parallel computation.

I would not choose between Myrinet and Gigabit Ethernet for large
cluster (> 32 nodes). I would rather make my decision between Myrinet
and Fast Ethernet. Fast Ethernet is cheap, switches scale much better
and some applications may not need more communication performance. Your
applications are the final judgment: if Fast Ethernet is enough (and
that's the case for some CPU-intensive-almost-embarrasingly-// code), go
for it. If your application needs a better pipe, Myrinet would be a
better choice, among others (Quadrics, SCI, ServerNet, etc).

Regards.

-- 
Patrick Geoffray

---------------------------------------------------------------
|      Myricom Inc       |  University of Tennessee - CS Dept |
| 325 N Santa Anita Ave. |   Suite 203, 1122 Volunteer Blvd.  |
|   Arcadia, CA 91006    |      Knoxville, TN 37996-3450      |
|     (626) 821-5555     |      Tel/Fax : (865) 974-0482      |
---------------------------------------------------------------