[Beowulf] SC|05: will 10Ge beat IB?

[Douglas Eadline]

> There is also the question as to whether GigE will be able to demonstrate
> the ultra low latencies seen in high performance interconnects such as
> Quadrics, InfiniBand and Myrinet.  In the end, there will most likely
> remain
> a market for high performance interconnects for high end applications.

Maybe not ultra low, but I have recently played with GAMMA
(http://www.disi.unige.it/project/gamma/) on some Intel
"workstation" GigE NICS. (The current version of GAMMA will only work on
Intel GigE chipsets)

For a 32 bit/33 Mhz PCI (Intel PRO/1000 MT RC82540EM) and a
SMC 8508T 8 port switch, I got the following (all measurements
are through the switch):

  Gamma ping-pong test gives latency as (MTU 4120):

    (lowest < average < highest)
    9.13639 < 9.54739 < 15.5897 us

    The maximum throughput was 72 MB/sec

  Netpipe for GAMMA MPI reported (MTU 4120):

    11 us latency
    68 MB/sec max throughput

  Netpitpe TCP (no GAMMA) reported

    29 us latency
    66 MB/sec  throughput (6000 byte MTU)

I have not had time to test this extensively and the above data does not
give a clear indication how GAMMA will help real codes, but it is
interesting how much performance you can pull out of a $30 NIC. I did run
some tests using HPL, but the next thing I want to try is the NAS suite
and see if it makes a big difference. Plus, I have not used it enough to
comment on stability.


-- 
Doug