[Beowulf] NFS + IB?

[Jeffrey Layton]

Dell has been doing some really good things around NFS and IB using IPoIB.
Their NSS storage solution is an example. They've gotten some pretty good
sequential performance: 2.25 GB/s for writes and 5.48 GB/s for reads.
Pretty good performance from a simple NFS server using IPoIB I think.

They publish the settings they use and also the tests they run. For more
information just Google for Dell NFS, NSS. There should be pdf's that are
downloadable.

The last results I can find are for NSS v5.5 which uses RHEL 6.4. No newer
kernels I'm afraid.

Enjoy!

Jeff


On Fri, Feb 20, 2015 at 7:15 PM, Bill Broadley <bill at cse.ucdavis.edu> wrote:

> I read through the beowulf archives for mentions of NFS + IB.  I found
> nothing
> newer than 2012.
>
> What are peoples current experience with NFS + IB?  I'm looking at the
> options
> for smaller clusters with /home on NFS.  I'll leave distributed
> filesystems for
> a separate discussion.
>
> The two leading options seem to be IPoIB and RDMA.  Any comments on
> stability
> and performance assuming a fairly recent kernel like 3.13 or newer?
>
> I'd be especially interested in any particular settings like rsize, wsize,
> mtu
> (2044 for datagram mode), connected mode vs datagram mode, and similar
> tuning
> parameters.  If you tuned for random I/O or bandwidth please mention it.
>
> I talked to one vendor that claims FDR has a larger than expected
> performance
> advantage over QDR for NFS.  Can anyone verify or guess as to exactly why?
>
> Not sure if the IPoIB tuning is relevant for NFS + RDMA.
>
> I've read these links, if there are other useful links please let me know:
>    https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs-rdma.txt
>    https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt
>
>
> http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf
>
> I'm happy to summarise the results if people prefer sending info directly
> to me.
>
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20150222/a20acb83/attachment.html>