[Beowulf] [beowulf] nfs vs parallel filesystems

[Janne Blomqvist]

On Sat, Sep 18, 2021 at 8:21 PM Lohit Valleru via Beowulf
<beowulf at beowulf.org> wrote:
>
> Hello Everyone,
>
> I am trying to find answers to an age old question of NFS vs Parallel file systems. Specifically - Isilon oneFS vs parallel filesystems.Specifically looking for any technical articles or papers that can help me understand what exactly will not work on oneFS.
> I understand that at the end - it all depends on workloads.
> But at what capacity of metadata io or a particular io pattern is bad in NFS.Would just getting a beefy isilon NFS HDD based storage - resolve most of the issues?
> I am trying to find sources that can say that no matter how beefy an NFS server can get with HDDs as backed - it will not be as good as parallel filesystems for so and so workload.
> If possible - Can anyone point me to experiences or technical papers that mention so and so do not work with NFS.
>
> Does it have to be that at the end - i will have to test my workloads across both NFS/OneFS and Parallel File systems and then see what would not work?
>
> I am concerned that any test case might not be valid, compared to real shared workloads where performance might lag once the storage reaches PBs in scale and millions of files.

For one thing NFS is not cache coherent, but rather implements a
looser form of consistency called close-to-open consistency. See e.g.
the spec at https://datatracker.ietf.org/doc/html/rfc7530#section-1.4.6

One case in which this matters is if you have a workload where
multiple nodes concurrently write to a shared file. E.g. with the
ever-popular IOR benchmarking tool, a slurm batch file like

#SBATCH -N 2                              # 2 nodes
#SBATCH --ntasks-per-node=1    # 1 MPI task per node
SEGMENTCOUNT=100
#Offset must be equal to ntasks-per-node
OFFSET=1
srun IOR -a POSIX -t 1000 -b 1000 -s $SEGMENTCOUNT -C -Q $OFFSET -e -i
5 -d 10 -v -w -r -W -R -g -u -q -o testfile

This should fail due to corruption within minutes if the testfile is
on NFS. Not saying any parallel filesystem will handle this either.
Some will.

-- 
Janne Blomqvist