Kidger's comments on Quadric's design and performance

[Steffen Persvold]

On Tue, 23 Apr 2002, Joachim Worringen wrote:
[snip]
> Richard Fryer wrote:
> > Also a brief note about the Dolphin product line, since the issue of link
> > saturation has come up:  - they DO also sell switches - or at least offer
> > them.  And if you check the SCI specification, you'll see that there are
> > some elaborate discussions of fabric architectures that the protocol
> > supports and switches enable.  What I DO NOT know is if the SCALI software
> > supports switch-based operation, and also don't know what the impact is on
> > the system cost per node.  My 'inexperienced' assessment of the appeal in
> > the Dolphin family is that you can start without the switch and later add it
> > if the performance benefit warrents.  That's what I'd say if I were selling
> > them anyway - and didn't know otherwise.  :-)
>
> The "external" switches are not designed for large-scale HPC
> applications (although they scale quite well inside the range of their
> supported number of nodes), but for high-performance, high-availabitlity
> small-scale cluster or embedded applications, as i.e. Sun sells. With
> ext. switches, you don't have to do anything to keep the network up if a
> node fails (and also nothing if it comes back as SCI is not
> source-routed). In torus topologies, re-routing needs to be applied to
> bypass bad nodes (Scali does this on-the-fly).
>
> Scali does not support external switches AFAIK (at least doesn't sell
> such systems any longer), which is less a technical issue but more a
> design-issue as the topology is fully transparent for the nodes
> accessing the network (they did use switches in the past, see
> http://www.scali.com/whitepaper/ehpc97/slide_9.html).
>

In theory there is no problem using the Scali SCI driver in a SCI switched
environment, we just haven't got the software to manage the switch (i.e
set up the routing tables). You could use Dolphin SW to manage the switch
though...

We used to use switches (and even the Dolphin driver) back in the SPARC
SBus days because (IIRC) the Dolphin SBus cards didn't have separate
out/in connectors (necessary to build ringlets and toruses).


> For large scale applications, distributed switches as in torus
> topologies scale better and more cost-efficient (see
> http://www.scali.com/whitepaper/scieurope98/scale_paper.pdf and other
> resources). With switches, you need *a lot* of cables and switches
> (which doesn't hinder Quadrics to do so - resulting in an impressive 14
> miles of cables for a recent system (IIRC) with single cables being up
> to 25m in length). It would need to be verified if such a system build
> with a Quadrics-like fat-tree topologie using Dolphins 8-port switches
> would scale better than the equivalent torus topologie for different
> communication patterns. I doubt it. At least, the interconect would cost
> a lot more (at least twice, or even more depending on the dimension of
> the tree).
>
> SCI-MPICH, can be used with arbitraries SCI topologies (because it uses
> the SISCI interface and thus runs with Scali or Dolphin SCI drivers). It
> is not that closely coupled to the SCI drivers as ScaMPI is.
>

It is true that ScaMPI uses a (proprietary) interface between userspace
(MPI library) and kernel space (SCI driver), but the SCI topology is
still transparent to the userspace layer.

Best regards,
Steffen