[Beowulf] FEC latency, was Re: [EXTERNAL] IB vs. Ethernet

[Lawrence Stewart]

> 
> So, in round numbers, it looks like you'd need to have ~500 bits to be able to run the decoder. If those are all in one lane at 100 Gbps - that's 5 nanoseconds.
> If it's split across 4 lanes at 25 Gbps, well, it's still 5 nanoseconds. 
> 
> That is comparable to the propagation delay in a 1 meter fiber.
> 

I thought I had posted about this a few weeks ago, but I can’t find it now…

Jim’s analysis is right, but unfortunately things are not that good.

The 100 GBPS Ethernet standard requires FEC, but you can get links without it.  There are at least four kinds.

DAC or Direct Attach Copper - these just drive twisted pairs, and add at most a few nanoseconds above the wire delay.  Only good up to about 5 meters

AOC or Active Optical Cable - sort of the same idea, but optical.  Each end adds about 2 nanoseconds to the fiber delay of about 5 ns/meter.  Usable up to 100 meters

PAM4 links - require some dsp work for the encoding and decoding, and evidently are about 10 ns/end.  I think this is some block scrambling.

FEC - As Jim says, the codes are Reed Solomon with 500 bits plus per block.  I found some transceiver specs which all cite 100-120 nanoseconds for the encoding and decoding.  

Sources:
Signal Integrity Journal: https://www.signalintegrityjournal.com/articles/1286-gbs-ethernet-forward-error-correction-fec-analysis

A thesis on the design and implementation of RS  coders for 100 Gbps: https://www.politesi.polimi.it/retrieve/a81cb05b-f419-616b-e053-1605fe0a889a/Report0.pdf

A powerpoint from IEEE about the latencies of the various FEC modes, there’s a nice table in here:
https://www.ieee802.org/3/dj/public/24_01/he_3dj_01_2401.pdf
he_3dj_01_2401
PDF Document · 660 KB

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260327/ca688a7c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: preview.png
Type: image/png
Size: 30582 bytes
Desc: not available
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260327/ca688a7c/attachment-0001.png>