[Beowulf] NFS + IB?

Jeffrey Layton laytonjb at gmail.com
Sun Feb 22 10:09:40 PST 2015

Dell has been doing some really good things around NFS and IB using IPoIB.
Their NSS storage solution is an example. They've gotten some pretty good
sequential performance: 2.25 GB/s for writes and 5.48 GB/s for reads.
Pretty good performance from a simple NFS server using IPoIB I think.

They publish the settings they use and also the tests they run. For more
information just Google for Dell NFS, NSS. There should be pdf's that are

The last results I can find are for NSS v5.5 which uses RHEL 6.4. No newer
kernels I'm afraid.



On Fri, Feb 20, 2015 at 7:15 PM, Bill Broadley <bill at cse.ucdavis.edu> wrote:

> I read through the beowulf archives for mentions of NFS + IB.  I found
> nothing
> newer than 2012.
> What are peoples current experience with NFS + IB?  I'm looking at the
> options
> for smaller clusters with /home on NFS.  I'll leave distributed
> filesystems for
> a separate discussion.
> The two leading options seem to be IPoIB and RDMA.  Any comments on
> stability
> and performance assuming a fairly recent kernel like 3.13 or newer?
> I'd be especially interested in any particular settings like rsize, wsize,
> mtu
> (2044 for datagram mode), connected mode vs datagram mode, and similar
> tuning
> parameters.  If you tuned for random I/O or bandwidth please mention it.
> I talked to one vendor that claims FDR has a larger than expected
> performance
> advantage over QDR for NFS.  Can anyone verify or guess as to exactly why?
> Not sure if the IPoIB tuning is relevant for NFS + RDMA.
> I've read these links, if there are other useful links please let me know:
>    https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs-rdma.txt
>    https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt
> http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf
> I'm happy to summarise the results if people prefer sending info directly
> to me.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20150222/a20acb83/attachment.html>

More information about the Beowulf mailing list