[Beowulf] TOE on Linux?

[John Hearns]

On Mon, 2008-05-19 at 18:42 -0400, Mark Hahn wrote:
> >
> > 1. Is having 10 GbE and Inifiniband in the same cluster overkill, or at
> > least unorthodox?  This cluster will be used by a variety of users
> 
> I would say so - if you've got IB, why add another interface?
> I'm not suggesting getting rid of gigabit, since its cost is 
> near-zero and ethernet _is_ the network.  OTOH, if there were a
> form of ethernet that competed with IB in price/latency/bandwidth,
> there would be no reason to go IB.
I back up what Mark says.

If you want to use 10Gig, put a 10Gig interface in your NFS server and
choose a switch with a suitable 10Gig port (or ports) plus 1gig ports.
Use the existing onboard 1gig interfaces on your nodes for the NFS
traffic.
You don't say what size, in number of compute nodes, your cluster will
be. It is very simplistic to think of simply throwing bandwidth at an
NFS solution and thinking this will solve everything - it won't.
Do you REALLY expect to saturate 1gig interfaces?