Large FOSS filesystems, was Re: [Beowulf] 512 nodes Myrinet cluster Challanges

[Bill Broadley]

> On a somewhat related note, are there any FOSS filesystems that can
> surpass 16 terabytes in a single filesystem - reliably?
> 
> Even something like a 64 bit linux system aggregating gnbd exports or
> similar with md or lvm and xfs or reiserfs (or whatever filesystem)
> would count, if it works reliably.

I've not used it, but I would suggest looking at Lustre (since pvfs2
has been already mentioned).  Personally I'm waiting for the Rocks roll
that is supposed to be released with 2 weeks of April 18th, alas it's
not out as far as I can tell.

I believe the meta-data server can fall over automatically to a warm
spare.  OST (where the data is stored) failures are usually handled by
attaching multiple OST's to the same disks.  Either via SCSI (as HP often
configures) or via fiber (often seen attached to DDN storage arrays).

So when an OST fails, one of the other (2 or 3) OSTs connected to the
same disks takes over (automatically).

I've never seen a good comparison of the functionality and performance
of PVFS2 vs Lustre, but I hope to explore that myself soon.

The GPL version of lustre used to be substantially older than the version
that was supported commercially, but recently they seem to be close
(both 1.4.X)

Corrections welcome, my 24TB isn't due for 4-6 weeks.

-- 
Bill Broadley
Computational Science and Engineering
UC Davis