[Beowulf] TOE on Linux?

[Mark Hahn]

> course not - Cisco sells both!), RDMA and TOE kept coming repeatedly as

the RDMA meme always puzzles me.  easy to see why it's a seductive 
idea, but I always run up against the problem of exactly how it's 
supposed to be used - are there any moderately well-known applications
that specifically use RDMA?  the closest I could find would be 
http://en.wikipedia.org/wiki/Global_address_space but I'm not sure 
whether it is really RDMA-specific...

> 1. Is having 10 GbE and Inifiniband in the same cluster overkill, or at
> least unorthodox?  This cluster will be used by a variety of users

I would say so - if you've got IB, why add another interface?
I'm not suggesting getting rid of gigabit, since its cost is 
near-zero and ethernet _is_ the network.  OTOH, if there were a
form of ethernet that competed with IB in price/latency/bandwidth,
there would be no reason to go IB.

> app" is meaningless. We're trying to build the best "one-size-fits-all"
> cluster to accomodate a very wide variety of applications.

IMO, that's an oxymoron.  it _is_ a good idea to have a quite general
configuration, but that inherently means that much capability will 
be wasted much of the time.  the only alternative I know is to reach
enough users to justify several different clusters, or at least sets
of nodes within one cluster.  it's entirely common to find pairs of 
users whose memory footprint, memory bandwidth, MPI bw or lat, disk
bw/latency/size demands each difer by 1000x.

> 2. I've read some about RDMA. Is it difficult to setup? What do I need
> to us it? Certain MPI implementations?  Certain kernel modules? Certain
> NICs/NIC drivers? A URL to a how-to would be sufficient. I'm sure I
> could find one on my own, but I'm interested in the discussion here.

afaik, RDMA is used by some MPI implementations; I don't know how common it
is to find Cray shmem put/get libraries any more.