[Beowulf] Gigabit Ethernet and RDMA

[Robert Kubrick]

CriticalIO has a silicon Ethernet NIC for GigE: http:// 
www.criticalio.com/XGE-Silicon-Stack-Ethernet.asp? 
gclid=CKvllaDwhpUCFQkcHgodOC1irA
An interesting recent article on GigE TOE:

TOE cards have fallen out of favor in recent years as server  
processors caught up to the task of processing Gigabit Ethernet.  
However, some observers predict TOE cards will be back as 10 Gigabit  
Ethernet network capacity leapfrogs processing power again.

http://searchstorage.techtarget.com/news/article/ 
0,289142,sid5_gci1322922,00.html



I think what Gus is asking is there exists some native *RDMA* support  
on GigE  NICs. I suspect most efforts in this area are concentrated  
on 10 Gig Ethernet. However, even if there were some GigE RDMA  
available, would you use the vendor API directly or need to fetch it  
through MPI?





On Aug 11, 2008, at 6:26 PM, Scott Atchley wrote:

> Hi Gus,
>
> Are you trying to find software for NICs you currently have? Or are  
> you looking for gigabit Ethernet NICs that natively support some  
> form of kernel-bypass/zero-copy?
>
> I do not know of any of the latter (do Chelsio or others offer 1G  
> NICs with iWarp?).
>
> As for the former, there are several options for cluster use:
>
> I believe Scyld has an optimized Ethernet stack. GAMMA has special  
> drivers for certain Intel NICs. PM/Ethernet-HXB is under active  
> development. Open-MX works over any Ethernet driver and works with  
> any MPI that works with native MX.
>
> If you are interested in 10G Ethernet, MX on Myricom 10G NICs can  
> work with our Myrinet switches or any brand of Ethernet switches  
> (some Ethernet switches provide lower latency than others).
>
> Scott
>
> On Aug 11, 2008, at 4:28 PM, Gus Correa wrote:
>
>> Hello Beowulf fans
>>
>> Does anyone know the status of  RDMA on Gigabit Ethernet?
>>
>> Is it a stable solution for a cluster interconnect, or still an  
>> experimental thing?
>>
>> Is it effective in offloading network tasks from the CPU?
>> (Myrinet and Infiniband seem to use RDMA effectively, right?)
>>
>> What does it take for it to work under typical Linux  
>> distributions? A driver?
>> A special kernel?
>> Something else? Just plug the NIC in and play?
>>
>> Does it support standard MPICH2 and/or OpenMPI compiled out of the  
>> box,
>> or does it require linking to some type of special low level  
>> communication library,
>> or perhaps requires the use of a special flavor or MPI (say, from  
>> the NIC vendor)?
>>
>> I poked around on the web,
>> and learned that Ammasso seems to have pioneered RDMA-enabled GigE  
>> NICs (Ammasso 1100).
>> Broadcom advertises a NIC with similar characteristics (BCM5706).
>> However, it is unclear if RDMA GigE NICs would work with standard  
>> Linux distros,
>> if it is effective, how much it costs, and how much hassle is  
>> required to make it work.
>>
>> Thank you,
>> Gus Correa
>>
>> -- 
>> ---------------------------------------------------------------------
>> Gustavo J. Ponce Correa, PhD - Email: gus at ldeo.columbia.edu
>> Lamont-Doherty Earth Observatory - Columbia University
>> P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
>> ---------------------------------------------------------------------
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit  
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf