[Beowulf] ipoib routing

[John Hearns]

Michael, the Mellanox gateway devices do exactly what you want.
What I implemented was a connection between an Infiniband HPC cluster and a
Data Lake system connected solely by Ethernet.

High level overview https://community.mellanox.com/docs/DOC-2301
http://www.mellanox.com/related-docs/prod_gateway_systems/PB_SX6036G.pdf

Look at this more up to date implementation though
http://www.mellanox.com/page/ib_fabricit_bxm_management

Drop me a mail off list if necessary.








On Tue, 11 Dec 2018 at 23:40, John Hearns <hearnsj at googlemail.com> wrote:

> Michael, yes.  Panasas engineered IPOIB to Ethernet routers for their
> storage platform. Remember that until the latest generation of their kit
> they ran on BSD, which had no Infiniband capability. Panasas IB routers
> booted from an onboard SATA DOM, which was quite a neat solution.
> I know this as I had to revive one from the dead after the technical staff
> at a company I worked with deleted the partition table during a test cycle
> (long war story there).
>
>
> More relevantly, when working at ASML I configured the Mellanox gateway
> devices which connect between IB networks and Ethernet networks.
> Technically these are bridges which implement the proxy arp function. I
> installed four of these devices, and got to know them pretty well.
> I think they do the job which you need.
>
>
>
>
>
>
>
>
> On Tue, 11 Dec 2018 at 17:03, Michael Di Domenico <mdidomenico4 at gmail.com>
> wrote:
>
>> does anyone use linux based routers to get from an ipoib network out
>> onto a regular ethernet network?  if so, do you have multiple routers
>> or just one?  in the case of multiple routers how to you handle the
>> traffic coming back from the ethernet network?  (ie how does the
>> ethernet network know which router to send the data back through)
>>
>> ---
>>
>> to expand further on what i'm trying to do (sanity/success is still to
>> be determine)...
>>
>> imagine a big cluster of machines all connected via infiniband running
>> ipoib
>> there are 4 linux routers on the ipoib network that also have 10Gb
>> connections to an ethernet network
>> the ethernet network contains a router to other resources
>>
>> what i want to accomplish is to have the compute nodes spray their
>> traffic across the 4 linux routers and end up on the ethernet network
>> i'm sure someone's done something like this before, but my google foo
>> is failing me
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20181211/253f2a3d/attachment.html>