[Beowulf] Slection from processor choices; Requesting Giudence

[Vincent Diepeveen]

a) the real big difference between highend and lowend nic's IMHO is
having DMA capabilities and the one way pingpong latency it gives.

Several persons replied and not a SINGLE ONE of them talks about
one way pingpong latency, which is one of the most important features
of a highend network for *many* applications.

It basically tells you in a primitive way: "my application at processor X 
needs some data
from some far away node in order to continue its calculation, what time do i 
need to
wait until i have that data?"

b) you quote your nic's there. Please give me the one way poingpong latency
from your NICs and also the PRICE of those nics. Not even asking after the 
price
of the fiber network, as the billions of dollars that lie in the underground 
is hard to
estimate anyway.

It's of course quite weird IMHO to pay 800 dollar for a 1 gigabit NIC,
if for a similar price you can pay some highend network card that has 10 
times more
bandwidth and a 50 times better one way pingpong latency.

If your argument is: "but it works for the software in question which works 
over tcp/ip and that
highend network card doesn't have a tcp/ip implementation", then that's a 
very weird
way of escaping as it doesn't change the outcome that there is a huge 
objective
difference between the highend network cards and the average $30 gigabit 
ethernet card.

Note that when i try to transfer i/o over default gigabit ethernet between 
some servers, using a
5 meters cross utp cable, that it's quite difficult to come above 20MB/s, 
this where both raid10
arrays handsdown handle speeds of a 100 MB/s (to say it polite). If you want 
to know the NIC
used. Just lookup which NICs the Tyan S2881 has.

This where further both servers were completely idle.

On the other hand at simplistic pci 64 bits 66Mhz (not exactly highend and 
quite dated) the
transfer between 2 dual k7's was 330MB/s with 2 quadrics cards (admittedly 
that wasn't
stored to disk but using the shmem).

Vincent

----- Original Message ----- 
From: "J. Andrew Rogers" <andrew at ceruleansystems.com>
To: "Vincent Diepeveen" <diep at xs4all.nl>
Cc: <beowulf at beowulf.org>
Sent: Saturday, June 17, 2006 12:27 AM
Subject: Re: [Beowulf] Slection from processor choices; Requesting Giudence


>
> On Jun 16, 2006, at 3:00 PM, Vincent Diepeveen wrote:
>> Jeff, we know how some people can mess up installs, but if you have 
>> gigabit ethernet,
>> with a one way pingpong latency of like 50-100 us if you're lucky,  which 
>> is not using DMA i
>> guess by default; in short nonstop interrupting your cpu, versus  newer 
>> myri cards that are probably under 3 us, factor 20 to 30  advantage to 
>> myri then you need to really not manage to install the  network to let 
>> gigabit ethernet work better.
>
>
> I get a round-trip latency between Linux servers (vanilla CentOS  install) 
> across a metro GigE IP network of ~140 us, which I guess  means we are 
> "lucky".  I would hope that a dedicated, tuned, and very  local GigE 
> network could do a lot better than what I get between  Internet servers 
> separated by kilometers of fiber and various bits of  network gear.  And I 
> know it can.
>
> You exaggerate the limitations of GigE.  Current incarnations of GigE  can 
> be close enough to Myrinet that it is perfectly functional for  some 
> applications and pretty competitive for someone willing to  extract the 
> capability from it.
>
>
> J. Andrew Rogers (who admittedly has an above average IP network)
>
>