[Beowulf] Parallel File system as a storage Solution

[Robert Latham]

On Thu, Aug 11, 2005 at 11:04:33AM -0400, Brian R Smith wrote:
> Hi all,
> 
> I just wanted to get everyone's thoughts on an idea we've been floating
> around here at our shop.

I'm one of the PVFS2 developers -- just warning you that I might be a
bit biased :>

> Lately, we've been investigating the possibility of using a centralized
> storage solution for our clusters.  We've looked at the high-priced
> SANs, iSCSI arrays, and a myriad of other solutions.  The problem is
> that given our funding, most of these options are not feasible.  We'd
> like to provide a common /home area for the users across all of our
> resources so that multiple copies of data are no longer scattered about
> multiple storage locations, wasting valuable disk space.

Typical /home directory usage patterns are quite different from
typical other file systems.  Users have lots of small files
(.dotfiles, directories, mail (if it's in maildir or mh format),
source code).  They run 'ls' (so end up calling stat(2) all the time.  

> We've been running PVFS2 as a system wide scratch space on a couple of
> our clusters and the stability has been rather surprising.  In the 8
> months since we've set up the system it has only gone down once because
> we had to take the system down for maintenance.  As well, we've had a
> number of people utilize this scratch area for running their jobs and
> there have been no issues so far.  

Glad to hear it.  we always hear about the problems, but sometimes
don't get to hear about the successful deployments.  You're using
PVFS2 exactly how we intended it

> Each I/O node on the PVFS2 setup has a mirrored 300GB volume for
> redundancy and I've written a number of utilities and scripts to ease
> management and to detect any problems that come up.  These have made for
> a fairly robust environment for our temporary storage needs but now we
> are wondering if this could be used on a more permanent basis.
> 
> Provided we have a reliable back-up system and hardware redundancy on
> our I/O servers, do you think it would be a "good idea" to investigate
> this further or am I just begging for trouble?

I'd have to say at this time that PVFS2 isn't a good chocie for home
directories.  You're free to try it, of course, but some operations
that are quite common for a /home directory -- and quite rare for
scientific applications -- such as compiling or unpacking a tarball
aren't as fast as they could be.  

I think you're on the right track here:  have an NFS-hosted /home
directory where users can compile their applications.  have a pvfs2
/scratch file system where those applications can write their IO when
they run.

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA                B29D F333 664A 4280 315B