[Beowulf] fast interconnects

[Gilad Shainer]

> > > 	- isn't Mellanox still the sole source for IB chips, for both
> > > 	nics and switches?  this seems odd if it's a thriving ecosystem.
> > > 	no offense intended!  yes, I know quadrics/SGI/SCI/Myri are all
> > > 	also sole-source.  but compared to the eth world...
> > 
> > Mellanox is not the sole source for InfiniBand chips. There are
other 
> > vendors like Qlogic and IBM that have InfiniBand chips. As
InfiniBand
>
> hmm, I know various people have produced IB chips, but I haven't seen
any in actual use.  oh, wait, qlogic - you mean pathscale infinipath.
> yes, certainly, though it's worth noting that the interesting part of
their product specifically just hijacks IB for their own faster MPI
protocol ;)
>
> not unlike octigabay/cray-xd1, come to think of it, and not really
_that_ far from the Horus folk using IB phy layers for their non-IB
fabric.
> 
> > > 	- price of nics has gone down significantly, close to the point
> > > 	of no resistance.  but afaikt, IB switching is still a big cost;
> > > 	are there prospects of IB switch fabrics getting anywhere close
> > > 	to gigabit costs?
> > 
> > Its all customers demand. The market pushes the price down and we
see 
> > the results already. Mellanox ASIC is $69 and adapter is $125 OEM
price.
> 
> what puzzles me is that market-push-down has apparenly only driven the
prices of nics down.  that's a bit silly, since surely shaving > $10 of
memory off a $100 nic hurts performance, especially when the majority of
the installation's cost is in switch+cable (at least > 5x the nic cost,
> no?)
>

In any of the testing I know of, there was no performance difference
between Memfree and nics with memory. Shaving the nics memory does not
only same some $ but also increase the reliability of your cluster.

>
> > > 	- IB people seem to be claiming quite nice latency figures 
> > > 	these days, but I don't really understand what changed to cause
> > > 	a ~7x reduction in latency.
> > 
> > In one word experience. In the early days, IB was not design for
high 
> > performance but for other reasons. But if you can get high BW and
low 
> > latency then why not... Mellanox increases the performance from 
> > generation to generation, and the latency we see today from multiple

> > vendors are quite nice figures..
>
> I'm afraid that upon reading this response and thinking about it, I
know no more than before ;(
>

You asked on the how come the latency has improved that much. It is all
about optimizations in both HW and SW....  And more to come in the near
future.

>
> > 10GigE people need to solve problems that do not exist in IB, and
the 
> > 10GigE solutions for those issues will results in more expensive
NICs.
> 
> but it's silly to only focus on nic prices.  I fully expect a myrinet
10gbase-t nic to cost at least around $500.  that sounds like a lot more
than a memfree IB nic, but if IB switch ports cost more than 10gbase-t
ports, it may be irrelevant.  besides, I'd actually prefer to pay a bit
more for my nics - to put intelligence in the nic rather than the
switch.
> 
>

I can not comment on the switch side, but I do believe the switch prices
become more attractive.