Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

Dan Stromberg strombrg at
Thu May 4 09:57:13 PDT 2006

> Ooops, sorry, english is not my native language and I can make
> mistakes :-) I liked pvfs before and I love pvfs2 now. 
> Well, I think the problems are those you are mentioning, first it 
> goes a bit slower than let's say nfs or something like gfs over gnbd
> (for small clusters)... in any case it is not so slow. The other
> is that you need the nodes that are metadata or I/O  servers have
> to be up, that means that the probability of file system failure is higher.
> The adventages are many, parallel I/O is a plus, not only for mpi programs
> but also for the normal tasks, if you  try to convert the format of a lot 
> of images you can split the work between nodes, but this is an adventage 
> only if your file system can handle that, which is not the case of nfs 
> obviously.
> In other words, pvfs2 is free, great and useful. it works well  as a 
> scratch area and it uses resources that otherwise are not visible
> for the user. And for myrinet users it goes over gm which is nice.

On a somewhat related note, are there any FOSS filesystems that can
surpass 16 terabytes in a single filesystem - reliably?

Even something like a 64 bit linux system aggregating gnbd exports or
similar with md or lvm and xfs or reiserfs (or whatever filesystem)
would count, if it works reliably.

More information about the Beowulf mailing list