[Beowulf] how large of an installation have people used NFS with? would 300 mounts kill performance?

[Rahul Nabar]

On Wed, Sep 9, 2009 at 2:12 PM, Bruno Coutinho <coutinho at dcc.ufmg.br> wrote:
> Because final NFS server bandwidth will be the bandwidth of the most limited
> device, be it disk, network interface or switch.
> Even if you have a switch capable of fill line capacity for all 300 nodes,
> you must put a insanely fast interface in your NFS server and a giant pool
> of disks to have a decent bandwidth if all nodes access NFS at the same
> time.

I'm thinking of having multiple 10GigE uplinks between the switch and
the NFS server. The actual storage is planned to reside on a box of
SAS disks. Approx 15 disks. THe NFS server is planned with at least
two RAID cards with multiple SAS connections to the box.

But that's just my planning. The question is do people have numbers.
What I/O throughputs are your NFS devices giving?  I want to get a
feel for what my I/O performance envelope should be like. What kind of
I/O gurrantees are available? Any vendors around want to comment?

On the other hand just multiplying NFS clients by their peak bandwidth
(300 x 1 GB) is an overkill. THat is a very unlikely situation. What
are typical workloads like? Given x NFS mounts in a computational
environment with a y GB uplink each what's the factor on the net
loading of the central storage? Any back of the envelope numbers?


> But depending on the way people run applications in your cluster, only a
> small set of nodes will access NFS at the same time and a Ethernet 10Gb with
> tens of disks will be enough.

strace profiling shows that app1 has very little NFS I/O. App2 has
about 10% runtime devoted to NFS I/O. Multiple seeks only. More reads
than writes. (All this thanks to Jeff Laytons excellent strace
analyzer and profiling help)

-- 
Rahul