[Beowulf] curiosity killed the cat

[John Hearns]

If anyone is interested in infrastructures using Ethernet like that,  have a look at JASMIN int he UK

http://www.ceda.ac.uk/projects/jasmin/
http://www.channelweb.co.uk/crn-uk/news/2339003/viglen-scents-hpc-success-with-gbp4m-jasmin-deal

Mellanox switching with 10Gbps to the edge and a rather large 40Gbps core infrastructure.
Put some of the switches in with my own fair hands!

________________________________________
From: Beowulf [beowulf-bounces at beowulf.org] on behalf of Stu Midgley [sdm900 at gmail.com]
Sent: 02 June 2016 05:01
To: Greg Lindahl
Cc: Beowulf List
Subject: Re: [Beowulf] curiosity killed the cat

just need to workout how to build large fat-trees with ethernet.

With 100G we get by with mlag for the spines (upto 640 hosts on 10g).

with our 40G/10G network, we are using ospf/mpls/vpls to give a static fat-tree...  30k lines of config applied to ~20 switches.  Nasty.

On Thu, Jun 2, 2016 at 11:57 AM, Greg Lindahl <lindahl at pbm.com<mailto:lindahl at pbm.com>> wrote:
On Thu, Jun 02, 2016 at 07:48:10AM +0800, Stu Midgley wrote:
> That's it.  As I said, I haven't used it since the early 00's.
>
> With 100Gb becoming common it might be time for these sort of MPI's to come
> back.

You could do better than these old interfaces, too... modern ethernet
chips have multiple send/receive queues, so you could hack up an
interface which looks a lot more like InfiniPath's one-queue-per-core
model -- the OpenIB code for InfiniPath also has code that maps M
cores:N queues if M>N.

-- greg




--
Dr Stuart Midgley
sdm900 at sdm900.com<mailto:sdm900 at sdm900.com>
Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP