From lindahl at pbm.com  Fri Mar  7 06:14:40 2025
From: lindahl at pbm.com (Greg Lindahl)
Date: Thu, 6 Mar 2025 22:14:40 -0800
Subject: [Beowulf] MPI over RoCE?
In-Reply-To: <8a32f2f1-9cb1-4940-b3d5-c2e16615755c@ucar.edu>
References: <475073c1-2d7a-45f8-bcfb-994e66a651bc@gmail.com>
 <8a32f2f1-9cb1-4940-b3d5-c2e16615755c@ucar.edu>
Message-ID: <20250307061440.GA23092@rd.bx9.net>

On Thu, Feb 27, 2025 at 12:28:11PM -0500, Prentice Bisbal wrote:

> I hope this isn't a dumb question: Do the Ethernet switches you're
> looking at have crossbar switches inside them?

No modern switch chip has a crossbar. They're rings, and the ring has
high enough capacity to work as well as a crossbar would.

-- greg



From prentice at ucar.edu  Fri Mar  7 21:53:34 2025
From: prentice at ucar.edu (Prentice Bisbal)
Date: Fri, 7 Mar 2025 16:53:34 -0500
Subject: [Beowulf] MPI over RoCE?
In-Reply-To: <20250307061440.GA23092@rd.bx9.net>
References: <475073c1-2d7a-45f8-bcfb-994e66a651bc@gmail.com>
 <8a32f2f1-9cb1-4940-b3d5-c2e16615755c@ucar.edu>
 <20250307061440.GA23092@rd.bx9.net>
Message-ID: <0b5d6e84-1be5-420c-a8de-61327f5a5aae@ucar.edu>


On 3/7/25 1:14 AM, Greg Lindahl wrote:
> On Thu, Feb 27, 2025 at 12:28:11PM -0500, Prentice Bisbal wrote:
>
>> I hope this isn't a dumb question: Do the Ethernet switches you're
>> looking at have crossbar switches inside them?
> No modern switch chip has a crossbar. They're rings, and the ring has
> high enough capacity to work as well as a crossbar would.

I don't think this is true anymore. I've been looking into this myself 
recently, and from what I've been told, some high-end Ethernet switches 
do have crossbar chips inside them. Unfortunately, I haven't been able 
to identify any models that do, but I'm also not a network engineer, so 
I'm not up with the features of all the different models.

Of course if they're only in the "high-end" or "high-performance" 
Ethernet switches, they probably cost as much or more than an IB switch.


-- 
Prentice