low-latency high-bandwidth OS bypass user-level messaging for commodity(linux) clusters with commodity NICs(<$200), HELP! (GAMMA/EMP/M-VIA/etc.)

[Michael T. Prinkey]

Hi Jon,

I began investigating the same issue a few months ago as Cu gigabit cards
and switches became widely available.  The situation is much the same now as
it was then.  No one is actively developing bypass support for the new
networking hardware.  The state of MVIA in particular has been in limbo for
at least 18 months.  GAMMA is interesting and seems to be more actively
developed, but when I last checked there was no support for SMP.  The MVIA
web site was promising a second-generation of the core code which intended
to make driver development more simple.  To my knowledge, that version 2 has
not seen the light of day.

I think that many people would love to take advantage of newer cheap gigabit
hardware with OS bypass, but as of yet, no one is really taking the lead in
pulling the drivers together.  In my mind, this is the next big hurdle for
the Beowulf community to overcome if we intend to really use commodity
hardware for network interconnects.

Mike Prinkey
Aeolus Research, Inc.

> Hi Donald Becker, master of all that is networking!  And anyone else
> that can help :)
>
> Perhaps this isn't the best way to get ahold of you, but I've also sent
> this to the Beowulf list.  I've noted your comments on OS Bypass
> drivers in the past.  But isn't there some room for non-TCP/IP related
> traffic, such as with computing clusters?  We don't need no stinking
> TCP!  No associated revenue?  You could replace Myrinet in the
> thousands of nodes we have here ALONE at NCSA.
>
> We (UIUC theoretical astrophysics group) are in the midst of purchasing
> a $50K cluster (I know, small, but big for us! :)) and I'm done all the
> research as to what we should be getting.  We ended up going with a
> Intel Desktop gigabit board and P4, but have found the tests to be very
> poor.  We only have 4 nodes right now because we worried about this
> very thing.
>
> Anyways, our problem is we are willing to pay for a commercial product,
> but not at any cost, perhaps upto $200 per board.  Basically, we see
> all these solutions such as:
>
> Giganet using VIA
> ServernetII using VIA
> InfiniBand
> U-Net
> AM II
> LPC
> PM
> FM
> GigaE-PM
> BIP
> EMP
> GAMMA
> M-VIA
>
> Half of these are seemingly dead, those that seem relatively alive are:
>
> M-VIA: http://www.nersc.gov/research/ftg/via/
> Only support a few devices, and only 1 expensive ($500) gigabit board
> that's still available (the SysKonnect).
>
> GAMMA: http://www.disi.unige.it/project/gamma/index.html
> Depending on what part of their website you are at, they support
> different devices.  The Alteon TIGON-II results seem to suck for
> latency, which is our biggest problem.  The Netgear GA621 looks great!
> But we already bought a $5000 copper gigabit switch!  We are stuck with
> it! (HP Procurve 5308xl).  Whether they support the GA622 is kinda open
> or at least untested according to the website.  No luck getting in
> touch with driver writer about that.  Besides, EMP guy says the GA622
> sucks!
>
>
> EMP: http://www.osc.edu/~pw/emp/
> Seems to be interesting, although the available 3Com 3C996, of which we
> have 3 to test, is only said to be "maybe" supported since it's Tigon 3
> and not Tigon 2.  And will it such in latency just like the Tigon 2?
> EMP guys says the GA-622T sucks with it's ns chipset and that was one
> option with GAMMA, assuming he really did write the driver for both 621
> and 622 (their website isn't clear about this, and no emails from the
> guys there), since the 622 "was" an option.
>
> Basically after all my testing (about 2 months of light testing and
> last 2 weeks of hard-core 24-hour a day testing) I realized TCP sucks
> and I need an OS Bypass or user-level communication driver.
>
> My questions are:
>
> 1) Is there a commercial product for a not so expensive board that
> provides what GAMMA/EMP/M-VIA provide?  Any other OS-bypass driver/MPI
> layer I don't know about?
>
> 2) Is there a solution I'm missing?  Has to be copper gigabit for
> linux, OS-bypass like GAMMA, MPI on top of that GAMMA-like.  No dead
> boards, etc.  Why are there no commercial products?  MPI/Pro is just a
> funny MPI still on top of TCP, no?
>
> I basically want 20us latency for 0 message size and peak bandwidth,
> for $100-$200 per board on gigabit.  Not too much to ask? :)   I know
> it's certainly possible.
>
> Currently with Myrinet on P4 I get 17us latency and 80MB/sec bandwidth,
> gigabit on P4 gets 70us latency and 80MB/sec.  On Xeon's I get 50us
> latency and 95MB/sec bandwidth with latest 3com bcm5700 or latest intel
> e1000 driver.
>
> I'm going to try EMP with my 3c996, but honestly his instructions are
> damn vague and confusing (i.e. WHAT snapshot of gcc/binutils should I
> use, what the heck do I do?, etc.)  Might try GAMMA too since EMP says
> it may work as a Tigon processor.  GAMMA seems a bit less crazy.
>
> Honestly, I can't really figure out what Scyld does.  Is it just a
> linux distribution?  Does it actually have OS-bypass networking?  Does
> anything?
>
> Why is the OS-bypass so hard?  If wanting no TCP support, isn't it
> easier than writing standard linux driver? (like you've done a lot!)
>
> Thanks!
> Jonathan McKinney
> University of Illinois at Urbana-Champaign
> Center for Theoretical Astrophysics
> NCSA
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf