[Beowulf] Can one Infiniband net support MPI and a parallel filesystem?

[Kozin, I (Igor)]

> Generally speaking, MPI programs will not be fetching/writing data
> from/to storage at the same time they are doing MPI calls so there
> tends to not be very much contention to worry about at the node level.

I tend to agree with this. 
 
> > Are there other practical and  cost effective alternatives to this
idea?
> 
> If the cluster is small enough, using gigabit with a shared filesystem
> is preferred since IB's low latency has relatively little affect on
> the big source of latency in any storage system: the physical disks.
> It's not until you cross the gigabit bandwidth barrier that IB really
> starts to make sense--and that's a barrier that's not crossed that
> often in a small cluster.

I was thinking whether it is practical to use compute nodes as storage
nodes thereby creating distributed fs on the very same cluster. 
The down side of this is much higher interference between apps running
in parallel even if the MPI and storage networks are physically
different.
But then the aggregate i/o bandwidth is going to be colossal even over
gigabit. Besides there are so many cores in modern servers that some can
be put aside if necessary with little or no performance penalty.
Such approach is certainly not a good idea for large clusters because
the jitter will kill all the scaling but perhaps given the right setup
and applications it should work well on a small cluster.
It turned out pivot3 has done something along this already
http://www.pivot3.com/