FNN vs GigabitEther & Myrinet

[Patrick Geoffray]

Hi,

Velocet wrote:

> (These questions apply to myrinet as well, obviously noting that its
> highly optimized and better performance than GbE but also more expensive.)

I am always surprised by this statement. It's natural because the GigE NICs 
are much (MUCH) less expensive than Myrinet's ones. However, the total 
price per node is another story for medium/large configuration: GigE switches 
are still (too) expensive for 32+ ports. I don't know if there is a technical 
reason or if the market is large enough to keep a high price tag.

> In the ArsTechnica article, the bandwidth per node of having 3 or 4
> fast Ether NICs in each node was said to be 'just as fast' as a single
> GbE NIC onboard. Obviously the GbE nic will not operate at 100% efficiency,
> which is related to the previous thread I just posted on.

I have never seen a GigE NIC pushing ~1 Gb/s. It's hard to push so much 
with IP, as the IP stack becomes quickly the real bottleneck. To reach the 
full hardware performance, you need an OS-bypass protocol, or a zero-copy 
IP stack. Folks from the project project at Duke used a such IP stack in 
FreeBSD to reach 2 Gb/s TCP over Myrinet. Without this special kernel and 
driver, the same hardware provides 1.2 Gb/s TCP with the Linux 2.4 IP stack.
It was way less efficient with 2.2 kernels.

> I was also wondering if there arent problems with having 4 or 5 NICs on
> a PCI bus - dont you run into shared IRQs which may result in delays on
> each node handling incoming traffic?

5 devices on the same PCI starts to be a problem: high number of interrupts, 
long arbitration loop on the PCI, short DMA transactions. It's OK with Fast 
Ethernet NICs as they are slow anyway.

> Obviously any bandwidth problems with the PCI bus itself will affect a GbE
> NIC too. 32bit 66MHz is technically 2Gb/s+ but you dont get that to
> the cards direclty Im sure. Can regular PCI even fill a GbE network?

I don't know any 32 bits/66 MHz PCI buses. It's usually 32/33, 
64/33 and 64/66. 32/33 has a peak of 1 Gb/s, so it's enough 
to fill up a GigE link, on the paper.

Regards

Patrick 

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      patrick at myri.com 
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------