[Beowulf]Infrastruture planning for small HPC 40/100 gigabyet eyhernet or Infiniband?

[Jim Lux]

At 09:33 AM 7/29/2008, Perry E. Metzger wrote:

>Eugen Leitl <eugen at leitl.org> writes:
> > On Sun, Jul 27, 2008 at 07:19:56PM -0700, Jim Lux wrote:
> >
> >> bear in mind that ordinary ethernet both coax and twisted pair is
> >> galvanically isolated.
> >
> > This is strange, because I've seen (small) sparks and received (mild)
> > shocks from both, in two different locations.
>
>Ground loops are a real phenomenon in UTP Ethernet. For example,
>*NEVER* run UTP between buildings. If the grounds in the two buildings
>are at a different relative potential, and they often are, very bad
>things can happen.

And for some interesting reasons...

First off, there's an isolation transformer as part of every UTP 
Ethernet interface, which one might think would solve the problem..

http://www.freescale.com/files/microcontrollers/doc/app_note/AN2759.pdf

So here's some specs.. 1500V isolation (ok, so you're probably not 
going to get outright breakdown).. and this is actually tested during 
manufacture, usually (HiPot testing)..


-40dB differential to common mode isolation..

What's this mean in practical terms?  let's just say that there's 
120VAC on the pair (that's common mode)... the transformer will 
isolate that from the other side by 40dB.. a factor of 100 in 
voltage, so now you're looking at 1.2V as a differential mode signal 
into the receiver.. oops.. that's more than enough to screw up the 
connection.  And, if the coupling to the two wires isn't the same, 
then you have a differential mode signal, which is coupled right on 
through the transformer.

(granted, the isolation spec is at 1 MHz, who knows what it might be 
at 50 or 60 Hz)

And, of course, if there is stray capacitance from UTP to the victim 
circuit, it could actually flow significant current.. 0.01 uF at 60 
Hz is about 260Kohms.. you get around a milliamp leakage current, 
which, granted, won't make a spark, but imposed across a 10K input 
impedance for a receiver amplifier will certainly cause troubles.

ESD is always an issue:
http://ieeexplore.ieee.org/iel5/10111/32405/01513539.pdf?arnumber=1513539
is an interesting paper looking at coupling between cables and such

And here's a ap note from Intel about transformerless interfaces
ftp://download.intel.com/design/network/applnots/ap438.pdf
It notes "We have developed a simple solution that can be used in a 
wide variety of such applications with the intent of simplifying the 
design cycle and reducing development time enabling products to enter 
the market in a shorter time frame than otherwise might be possible "

On the other hand, they also warn:Magnetic-less LAN designs should 
not be done when the LAN signals must be routed through ables that 
are external to the system chassis. The isolation transformer in a 
magnetics module, provides some level of improved safety in the event 
that higher voltages or ESD gets onto the LAN cable. Magnetic-less 
LAN designs should only be done when the differential circuits or 
cables will be routed internal to the same chassis.

So, "within the rack" in a cluster might be able to use these techniques.

In any event, the design shown in the ap note basically uses 0.056 uF 
in series with each wire of the pair, with a 0.1 uF to 
"chassis"...  That 0.1 uF is only 26K at 60Hz, so if you (foolishly) 
used one of these designs to connect between two buildings where the 
chassis are at, say, 50V differential, that's a BIG problem.






>The building complex I live in ran Cat 5 between buildings in
>underground ducts. They were very surprised when lightning strikes
>some distance away regularly blew out the switches. Changing to fiber
>eliminated the problem, of course.
>
> > In any case I'll have an electrician diagnose the problem.
>
>If you're seeing sparks, as you say, I suspect you do indeed have an
>AC supply problem. Ground loop, or something worse. (The Electrical
>Wiring FAQ describes several problems that qualify as "worse"...)
>
>--
>Perry E. Metzger                perry at piermont.com