[Beowulf] Storage recommendations?

[Robert Horton]

On Sun, 2009-11-01 at 22:25 -0500, Joe Landman wrote:
> Robert Horton wrote:
> > Hi,
> > 
> > I'm looking for some recommendations for a new "scratch" file server for
> > our cluster. Rough requirements are:
> > 
> > - Around 20TB of storage
> > - Good performance with multiple nfs writes (it's quite a mixed workload
> > so hard to characterise further)
> > - Data security not massively important as it's just for scratch /
> > temporary data.
> > 
> > It'll just be a single server serving nfs, I'm not looking to go down
> > the luster / pvfs route.
> 
> 
> a) what network fabric (IB, 10GbE, GbE, ...)

Currently using the GigE network for the NFS traffic. There is a DDR
Infiniband network in place for the MPI traffic which could potentially
be used for storage, however my impression is that the current
bottleneck is the disk io (or iops) rather than network, so I'm not sure
that this would be worth the mucking about.

> b) roughly how many simultaneous writers ... large block streaming, or 
> small block random?  How much sustained IO (MB/s) do you need to support 
> your worker machines?

The jobs typically use 32 nodes with 8 processes on each node, so I
guess ~250. Averaged over 5 minutes, maximum IO is around 15MB/s read
and 0.5MB/s for write, however I've seen short periods of around 30MB/s
write.

The problem, basically, is that under certain heavy IO conditions which
I haven't managed to reliably reproduce, the whole machine becomes very
slow, in some cases making interactive work pretty much impossible.
Hence the desire to move the storage off the headnode.

> 
> c) looking to build it your self or buy units that work?
> 

Preferable the second option, although I've not completely ruled out
building something.