[Beowulf] IB vs. Ethernet

[Lawrence Stewart]

Well I am in the “low latency is actually important” group.  IB usually has dramatically lower latency than Ethernet, but neither are very impressive.

In 1995, the Cray T4D had 1000 nodes and a 4 Gbps mesh network, and put latencies of 800 ns or so.  Switch forwarding was, IIRC, 40 ns straight or 160 to turn a corner.

In 2025, IB has put latencies of … 1 microsecond, in spite of having 200 Gbps or 400 Gbps links.

Slingshot is really excellent for congestion control, but end to end latency on Aurora is about 2.3 microseconds.  Switch forwarding time is 300 ns (!)

Ultra Ethernet is paying at least lip service to latency, as in the slides mention it, but there are no numbers that I have seen.

My own opinion is that we should really expect 300 ns end to end.  The links are fine.  The problems are the switches and the CPUs and PCIe.

CXL is supposed to be a substantial latency improvement over PCIe, but we’re talking like 90 ns rather than 150.  That is hardly enough to close the latency gap.

I think a 64 byte store at a core should directly become a packet.  No on-die-network, no coherence, no root complex, no host-fabric adapter.  Incoming short messages should be delivered directly to a fifo in the relevant core.

I don’t think any of this requires rocket science, just people to write the requirements properly.

-Larry

> On Jan 15, 2026, at 11:07, Prentice Bisbal <prentice at ucar.edu> wrote:
> 
> Beowulfers,
> 
> Here's a good topic for discussion: What are your thoughts on Ethernet vs. InfiniBand? Can UltraEthernet compete with/surpass IB, or is IB still the standard for non-proprietary (Slingshot, Tofu, etc.) networking for HPC? Discuss!
> 
> -- 
> Prentice
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf