Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges
strombrg at dcs.nac.uci.edu
Fri May 5 11:00:26 PDT 2006
On Thu, 2006-05-04 at 12:16 -0600, Craig Tierney wrote:
> Dan Stromberg wrote:
> >> Ooops, sorry, english is not my native language and I can make
> >> mistakes :-) I liked pvfs before and I love pvfs2 now.
> >> Well, I think the problems are those you are mentioning, first it
> >> goes a bit slower than let's say nfs or something like gfs over gnbd
> >> (for small clusters)... in any case it is not so slow. The other
> >> is that you need the nodes that are metadata or I/O servers have
> >> to be up, that means that the probability of file system failure is higher.
> >> The adventages are many, parallel I/O is a plus, not only for mpi programs
> >> but also for the normal tasks, if you try to convert the format of a lot
> >> of images you can split the work between nodes, but this is an adventage
> >> only if your file system can handle that, which is not the case of nfs
> >> obviously.
> >> In other words, pvfs2 is free, great and useful. it works well as a
> >> scratch area and it uses resources that otherwise are not visible
> >> for the user. And for myrinet users it goes over gm which is nice.
> > On a somewhat related note, are there any FOSS filesystems that can
> > surpass 16 terabytes in a single filesystem - reliably?
> What do you want to do with your 16 TB? Does PVFS2 not meet your needs
> or your level of reliability? What don't you find reliable about it?
We want to store scientific datasets - and we actually wanted more like
30T, but had to settle for less.
> I expect that xfs would work just fine. The question is, how can you
> access it? You can export it with NFS, but the performance doesn't scale.
If it'd be at least semi reliable, this application would probably be
fine with that.
> Why FOSS (not to start a flame war)? What if AcmeFS was reasonably
> priced and did what you needed it to do?
We already have a commercial solution that's working pretty well, so if
we go commercial, we might return to that. FOSS tends to improve faster
though, and if you get it with a support contract, it seems pretty
More information about the Beowulf