Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

[Dan Stromberg]

> Ooops, sorry, english is not my native language and I can make
> mistakes :-) I liked pvfs before and I love pvfs2 now. 
> 
> Well, I think the problems are those you are mentioning, first it 
> goes a bit slower than let's say nfs or something like gfs over gnbd
> (for small clusters)... in any case it is not so slow. The other
> is that you need the nodes that are metadata or I/O  servers have
> to be up, that means that the probability of file system failure is higher.
> 
> The adventages are many, parallel I/O is a plus, not only for mpi programs
> but also for the normal tasks, if you  try to convert the format of a lot 
> of images you can split the work between nodes, but this is an adventage 
> only if your file system can handle that, which is not the case of nfs 
> obviously.
> 
> In other words, pvfs2 is free, great and useful. it works well  as a 
> scratch area and it uses resources that otherwise are not visible
> for the user. And for myrinet users it goes over gm which is nice.

On a somewhat related note, are there any FOSS filesystems that can
surpass 16 terabytes in a single filesystem - reliably?

Even something like a 64 bit linux system aggregating gnbd exports or
similar with md or lvm and xfs or reiserfs (or whatever filesystem)
would count, if it works reliably.