Cables

[Pete Beckman]

At 04:20 PM 6/5/99 -0700, Keith Murphy wrote:
>If you are looking for (800MBytes) aggregate performance try SCI (Scaleable
>Coherent Interface) which also has an  extremely low latency (better than
>Myrinet).

I don't believe that is true.  Measuring latency is a place where poor
benchmarks and clever tricks abound.  Unfortunately, there is no standard
definition of "latency", just like some people say a Flop is a multiply and
add, and some people count a multiple and an add as two flops.   SCI
publishes that latency could be as low as 2.3 microseconds.  The thing to
remember is that all of these high-performance cards (HiPPI, Myrinet, SCI)
are generally limited first by the PCI bus.  It often takes nearly a
microsec to get anything from memory to the PCI bus and out the interface
of the card.  Then, ignoring time in flight, it takes about another
microsecond to get the data from the interface card over the PCI bus to the
system memory.  That's nearly 2 microseconds right there. 

Now here comes the tricky part... measuring latency.  How do you time it?
How do you know when the data arrived?  Do you have the remote CPU spin
wait on a memory location?  Does the card raise an interrupt (slow)?  What
do you call latency?  The time it takes for the sender to initiate the
transfer until the data has arrived, or until the data has been detected to
arrive?  I've seen people measure it both ways.  Does the data have to be
page aligned?  Word aligned?  Some interfaces have problems with unaligned
data.  Does the benchmark include a switch, or for the benchmark have the
two machines been plugged together with the equivalent of a cross-over
cable?  Does the latency benchmark time an unreliable send (no checksum to
detect message corruption) or does the interface card buffer and then check
the integrity of the data before putting it in memory?

Anyway, as you can see, head-to-head latency tests rarely happen.  The only
fair number is run a standard MPI latency ping/pong test, and report those
latencies.  Unfortunately, people rarely post their MPI latency measuring
code, so we can't even do head to head comparisons there.   The SCALI web
pages say: "ScaMPI's message latency, measured as half the round-trip delay
of a zero length MPI message, is less than 10 µsec".  The BIP folks in
France have an MPI over BIP that they report has a latency of 12 µsec, and
a bandwidth of 1 Gb/sec for a 8MB message.  As you can see, apples to
apples is hard.

-Pete



---
========================================================================
| Peter H. Beckman                   |  Advanced Computing Laboratory  |
|  Los Alamos National Laboratory    |     Phone: 505-665-0800         |
|  CIC/ACL MS-B287                   |     Fax:   505-665-4939         |
|  Los Alamos, NM 87545              |     email: beckman@acl.lanl.gov |
========================================================================