[Beowulf] Can one Infiniband net support MPI and a parallel file system?

[andrew holway]

Gus,

It works(ish) and people are doing it but my research has shown that
it is not yet stable. I have been talking to various companies
offering lustre support. They have all told me that they can do it but
none have been able to offer a reference site.

I bet you the chaps behind roadrunner aren't going to be publishing
any downtime figures.

as mentioned by mark, If you try and force lots of stuff down the
tubes you are going to break something. I guess its a _bit_ like
torrents on a naff home router, lots and lots of small torrent
connections filling up the nat table which cannot purge itself fast
enough which mean that larger downloads time out as they fall off the
bottom of the table.

Data on how hard IB switches have to work would be interesting. I have
a feeling that many people are taking their fabric to the very edge
and back again!

Perhaps someone can shed more light on the problems?

ta

Andy

On Tue, Aug 5, 2008 at 10:25 PM, Gus Correa <gus at ldeo.columbia.edu> wrote:
> Hello Beowulf fans
>
> Is anybody using Infiniband to provide both
> MPI connection and parallel file system services on a Beowulf cluster?
>
> I thought to have a storage node that would
> serve a parallel file system to the beowulf nodes over IB
> (something like a NFS on steroids).
> The same IB net would also work as the MPI interconnect.
>
> Is this design possible?
>
> On a small cluster, does it require two separate IB physical networks (cards
> and switch),
> or can it be done with a single IB card per node and one switch?
>
> Is this design efficient?
>
> Are there other practical and  cost effective alternatives to this idea?
>
> Would this type of design work with GigE instead of IB?
>
> I confess I know nothing about parallel file systems and IB.
> So, please forgive me if my questions are nonsense.
>
> I also appreciate any links to readings that would mitigate my ignorance
> on these subjects.
>
> Thank you,
> Gus Correa
>
> --
> ---------------------------------------------------------------------
> Gustavo J. Ponce Correa, PhD - Email: gus at ldeo.columbia.edu
> Lamont-Doherty Earth Observatory - Columbia University
> P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>