[Beowulf] test network link quality?

[David Mathog]

> we've noticed that occasionally cables of dubious quality
> pop up and cause issues with packet flow in either one or both
> directions.

That's one sort of error that would be nice to be able to detect,
especially in cases where the error is subtle.
 
> Using iperf we can see nodes that don't perform properly,
> and we notice a lot of problems that don't show up with simple ping
> tests.

Iperf is not exactly what I was looking for, but close.  An iperf report
looks like:

[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec  0.002 ms    0/  893 (0%)

For comparison of (low) error rates, all that's useful is the 0/893. 
The numbers in this run are too small to distinguish between 99.99 and
99.9999, and there is probably no variation in packet size or content to
help push the test to the edge.

Think of this another way - imagine you want to do quality tests on a
bunch of cables, and you don't have the proper test tools to make direct
electrical measurements.  You might be comparing samples between
different brands, or looking for slightly defective cables in a large
batch.  What program could you run that would give a useful metric for
transmission quality?  Granted, it may be that all of the cables are
"more than good enough", so that there is no correspondence between
cable quality and dropped/corrupted packets, in which case no difference
could be measured.  That's an OK outcome.  But if there is such a
correspondence, and it is small, what tool would you use to see it?  One
that would at the end of the process allow you to say "this cable is
unacceptable because it measures >4 sigmas below the mean for the
batch".  Such a cable would probably be perfectly acceptable for a
desktop connection, it just isn't the one you want shuffling data day
and night in a cluster.    (Same argument for NICs and switch ports.)

Thanks,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech