Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

[Craig Tierney]

Dan Stromberg wrote:
>> Ooops, sorry, english is not my native language and I can make
>> mistakes :-) I liked pvfs before and I love pvfs2 now. 
>>
>> Well, I think the problems are those you are mentioning, first it 
>> goes a bit slower than let's say nfs or something like gfs over gnbd
>> (for small clusters)... in any case it is not so slow. The other
>> is that you need the nodes that are metadata or I/O  servers have
>> to be up, that means that the probability of file system failure is higher.
>>
>> The adventages are many, parallel I/O is a plus, not only for mpi programs
>> but also for the normal tasks, if you  try to convert the format of a lot 
>> of images you can split the work between nodes, but this is an adventage 
>> only if your file system can handle that, which is not the case of nfs 
>> obviously.
>>
>> In other words, pvfs2 is free, great and useful. it works well  as a 
>> scratch area and it uses resources that otherwise are not visible
>> for the user. And for myrinet users it goes over gm which is nice.
> 
> On a somewhat related note, are there any FOSS filesystems that can
> surpass 16 terabytes in a single filesystem - reliably?

What do you want to do with your 16 TB?  Does PVFS2 not meet your needs 
or your level of reliability? What don't you find reliable about it?

I expect that xfs would work just fine.  The question is, how can you 
access it?  You can export it with NFS, but the performance doesn't scale.

Why FOSS (not to start a flame war)?  What if AcmeFS was reasonably 
priced and did what you needed it to do?

Craig





> 
> Even something like a 64 bit linux system aggregating gnbd exports or
> similar with md or lvm and xfs or reiserfs (or whatever filesystem)
> would count, if it works reliably.
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf