[Beowulf] Intel pulls networking onto Xeon Phi

[Greg Lindahl]

On Mon, Dec 02, 2013 at 08:41:26AM -0500, atchley tds.net wrote:
> On Mon, Dec 2, 2013 at 8:37 AM, atchley tds.net <atchley at tds.net> wrote:

> > I am not sure what Aries currently offers that IB does not.

The IB in question is the True Scale adapter, which does some things
really fast and other things pretty slowly. Aries has different
features (quite different and more capable than IB, really), and is
much larger.

To put this into perspective, I suspect the typical modern Ethernet
adapter has more gates than True Scale. If you're going to add
something to a CPU, it had best be small. CPU guys get really irate if
you reduce their yield.

> > As Myricom showed with MX over Ethernet followed by
> > Mellanox with RoCE, you can get low latency over Ethernet bypassing the
> > kernel and the TCP stack.

Indeed, Myricom+MX is quite similar in concept to the IB extension
found in True Scale. The main difference (in my mind) is that OpenMX
is hosted on a not-optimized-for-MX generic ethernet adapter, and that
Myrinet's hardware was not fully optimized to do exactly what MX
needed, nothing more and nothing less. True Scale is the smallest
possible adapter that supports the basics of what MPI needs.

The fabric is pretty irrelevant, as long as it has flexible
routing. (See below for comments about SDN.)

> > HPC sends a lot of small messages and various stacks are making use of
> > 8-byte atomics. It is unhelpful to have a 64 byte minimum frame size in
> > this case.

Yes, a smaller frame size is quite nice for achieving high message
rates for tiny packets.

> > Ethernet topology discovery protocols were designed for environments where
> > equipment can be changed out, expanded, or otherwise altered.

This has changed in the new SDN (software defined networking)
world. You can think of SDN on Ethernet as Infiniband management
protocols implemented in ethernet, making many of the same mistakes
that Infiniband did, plus some new ones.

> Ethernet requires a single-path between any two endpoints.

This is not true. It's more accurate to say that ethernet (especially
TCP) benefits from in-order delivery, which you can ensure either on a
host-host basis (which is what spanning tree provides) or on a
per-flow basis (which is what SDN allows.)

Personally, I'm a bit bummed this won't happen until 2015 :-( but I'm
really excited to see True Scale's basic design continue into another
generation.

-- greg