[Beowulf]Infrastruture planning for small HPC 40/100 gigabyet eyhernet or Infiniband?
James.P.Lux at jpl.nasa.gov
Tue Jul 29 10:10:21 PDT 2008
At 09:33 AM 7/29/2008, Perry E. Metzger wrote:
>Eugen Leitl <eugen at leitl.org> writes:
> > On Sun, Jul 27, 2008 at 07:19:56PM -0700, Jim Lux wrote:
> >> bear in mind that ordinary ethernet both coax and twisted pair is
> >> galvanically isolated.
> > This is strange, because I've seen (small) sparks and received (mild)
> > shocks from both, in two different locations.
>Ground loops are a real phenomenon in UTP Ethernet. For example,
>*NEVER* run UTP between buildings. If the grounds in the two buildings
>are at a different relative potential, and they often are, very bad
>things can happen.
And for some interesting reasons...
First off, there's an isolation transformer as part of every UTP
Ethernet interface, which one might think would solve the problem..
So here's some specs.. 1500V isolation (ok, so you're probably not
going to get outright breakdown).. and this is actually tested during
manufacture, usually (HiPot testing)..
-40dB differential to common mode isolation..
What's this mean in practical terms? let's just say that there's
120VAC on the pair (that's common mode)... the transformer will
isolate that from the other side by 40dB.. a factor of 100 in
voltage, so now you're looking at 1.2V as a differential mode signal
into the receiver.. oops.. that's more than enough to screw up the
connection. And, if the coupling to the two wires isn't the same,
then you have a differential mode signal, which is coupled right on
through the transformer.
(granted, the isolation spec is at 1 MHz, who knows what it might be
at 50 or 60 Hz)
And, of course, if there is stray capacitance from UTP to the victim
circuit, it could actually flow significant current.. 0.01 uF at 60
Hz is about 260Kohms.. you get around a milliamp leakage current,
which, granted, won't make a spark, but imposed across a 10K input
impedance for a receiver amplifier will certainly cause troubles.
ESD is always an issue:
is an interesting paper looking at coupling between cables and such
And here's a ap note from Intel about transformerless interfaces
It notes "We have developed a simple solution that can be used in a
wide variety of such applications with the intent of simplifying the
design cycle and reducing development time enabling products to enter
the market in a shorter time frame than otherwise might be possible "
On the other hand, they also warn:Magnetic-less LAN designs should
not be done when the LAN signals must be routed through ables that
are external to the system chassis. The isolation transformer in a
magnetics module, provides some level of improved safety in the event
that higher voltages or ESD gets onto the LAN cable. Magnetic-less
LAN designs should only be done when the differential circuits or
cables will be routed internal to the same chassis.
So, "within the rack" in a cluster might be able to use these techniques.
In any event, the design shown in the ap note basically uses 0.056 uF
in series with each wire of the pair, with a 0.1 uF to
"chassis"... That 0.1 uF is only 26K at 60Hz, so if you (foolishly)
used one of these designs to connect between two buildings where the
chassis are at, say, 50V differential, that's a BIG problem.
>The building complex I live in ran Cat 5 between buildings in
>underground ducts. They were very surprised when lightning strikes
>some distance away regularly blew out the switches. Changing to fiber
>eliminated the problem, of course.
> > In any case I'll have an electrician diagnose the problem.
>If you're seeing sparks, as you say, I suspect you do indeed have an
>AC supply problem. Ground loop, or something worse. (The Electrical
>Wiring FAQ describes several problems that qualify as "worse"...)
>Perry E. Metzger perry at piermont.com
More information about the Beowulf