[Beowulf] Questions regarding interconnects

[Richard Walsh]

Olli-Pekka Lehto wrote:

> Hello,
>
> I'm writing a paper on current and emerging cluster interconnect 
> technologies as a part of my University studies. I have included 1GbE 
> (incl. RDMA), 10GbE, Quadrics, InfiniBand and Myrinet. The goal is to 
> provide an introduction to the subject maybe more from a network 
> engineer's point of view with an overview on the key features and the 
> pros/cons of each solution. I have some questions on which I hope you 
> could help me out with:

    I think that integrating a custom interconnect for comparison into 
you analysis would be useful to contrast
    the capabilities of "commodity cluster" interconnects with those of 
the presumptive custom leading edge.
    I would choose the Cray X1e or Altix interconnects for this.

>
> What do you see as the key differentiating factors in the quality of 
> an MPI implementation? This far I have come up with the following:
> -Completeness of the implementation
> -Latency/bandwidth
> -Asynchronous communication
> -Smart collective communication

    I think that explicit treatment/comparison of the interconnect's 
RDMA capabilities is important as they support
    both MPI-2 and the new-ish UPC and CAF compilers for cluster 
systems.  I can send you a recent article I wrote
    comparing Quadrics to the Cray X1 interconnect relative to the 
performance of these global address space programming
    models (UPC and CAF).

    Another thing to look at is the latency advantage/potential of 
alternative paths to the processor (i.e HT/Infinipath)

>
> Are there any NICs on the market which utilize the 10GBase-CX4 
> standard and if there is are there any clusters which use them? Do you 
> see it as a viable choice for an interconnect considering the 
> relatively low cost of InfiniBand and that fact that 10GBase-T is not 
> that far in the future?
>
> When do you estimate that commodity Gigabit NICs with integrated RDMA 
> support will arrive to the market? (or will they?)


   AMASSO already sells one.

>
>
> best regards,
> Olli-Pekka



Richard Walsh
AHPCRC