[Beowulf] Slection from processor choices; Requesting Giudence
diep at xs4all.nl
Fri Jun 16 18:36:22 PDT 2006
a) the real big difference between highend and lowend nic's IMHO is
having DMA capabilities and the one way pingpong latency it gives.
Several persons replied and not a SINGLE ONE of them talks about
one way pingpong latency, which is one of the most important features
of a highend network for *many* applications.
It basically tells you in a primitive way: "my application at processor X
needs some data
from some far away node in order to continue its calculation, what time do i
wait until i have that data?"
b) you quote your nic's there. Please give me the one way poingpong latency
from your NICs and also the PRICE of those nics. Not even asking after the
of the fiber network, as the billions of dollars that lie in the underground
is hard to
It's of course quite weird IMHO to pay 800 dollar for a 1 gigabit NIC,
if for a similar price you can pay some highend network card that has 10
bandwidth and a 50 times better one way pingpong latency.
If your argument is: "but it works for the software in question which works
over tcp/ip and that
highend network card doesn't have a tcp/ip implementation", then that's a
way of escaping as it doesn't change the outcome that there is a huge
difference between the highend network cards and the average $30 gigabit
Note that when i try to transfer i/o over default gigabit ethernet between
some servers, using a
5 meters cross utp cable, that it's quite difficult to come above 20MB/s,
this where both raid10
arrays handsdown handle speeds of a 100 MB/s (to say it polite). If you want
to know the NIC
used. Just lookup which NICs the Tyan S2881 has.
This where further both servers were completely idle.
On the other hand at simplistic pci 64 bits 66Mhz (not exactly highend and
quite dated) the
transfer between 2 dual k7's was 330MB/s with 2 quadrics cards (admittedly
stored to disk but using the shmem).
----- Original Message -----
From: "J. Andrew Rogers" <andrew at ceruleansystems.com>
To: "Vincent Diepeveen" <diep at xs4all.nl>
Cc: <beowulf at beowulf.org>
Sent: Saturday, June 17, 2006 12:27 AM
Subject: Re: [Beowulf] Slection from processor choices; Requesting Giudence
> On Jun 16, 2006, at 3:00 PM, Vincent Diepeveen wrote:
>> Jeff, we know how some people can mess up installs, but if you have
>> gigabit ethernet,
>> with a one way pingpong latency of like 50-100 us if you're lucky, which
>> is not using DMA i
>> guess by default; in short nonstop interrupting your cpu, versus newer
>> myri cards that are probably under 3 us, factor 20 to 30 advantage to
>> myri then you need to really not manage to install the network to let
>> gigabit ethernet work better.
> I get a round-trip latency between Linux servers (vanilla CentOS install)
> across a metro GigE IP network of ~140 us, which I guess means we are
> "lucky". I would hope that a dedicated, tuned, and very local GigE
> network could do a lot better than what I get between Internet servers
> separated by kilometers of fiber and various bits of network gear. And I
> know it can.
> You exaggerate the limitations of GigE. Current incarnations of GigE can
> be close enough to Myrinet that it is perfectly functional for some
> applications and pretty competitive for someone willing to extract the
> capability from it.
> J. Andrew Rogers (who admittedly has an above average IP network)
More information about the Beowulf