[Beowulf] Ethernet network opinions

[Saravanaraj Ayyampalayam]

Hi Everybody,

Our network engineering is asking us to refresh out ethernet network, replace all the out of support top of rack switches and our core aggregator switches.

Our existing ethernet network is a flat layer 2 network with a few vlans. We have a hybrid QDR & EDR IB fabric for high speed internodal and Lustre traffic.
We have an isolated 100GbE ethernet for our Panasas storage and transfer nodes.

We are thinking of taking this opportunity to redesign our ethernet network. We are hoping to make managing the network easier, use automation and design a flexible network with this redesign.
One of the options we are looking at is a full layer 3 network using Cumulus Linux and Mellanox/Edgecore hardware.

I was wondering if anyone here has experience with Cumulus Linux and layer 3 networking that they can share?

Thank you for your time.
-Raj