[Beowulf] About TofuD in A64FX and Infiniband HDR

[Mikhail Kuzminsky]

My initial questions about bandwidth in a cluster:
I want to understand the bandwidth when using TofuD in A64FX or
Infiniband HDR. With HDR everything is clear - 25 GB / s for 4 links
or 75 GB / s for 12 links. But PCIe-v3 x16 in A64FX will keep
bandwidth.

Everything is more cunning with TofuD. Each link has 2 lanes, 28.05
Gbps x 2 gives about 7 GB / s (more really 6.8 GB / s).
6 TNI or 6 links give a total of 40.8 GB / s for injection - that's
more than 25 GB / s in 4x HDR. This is for 2.2 GHz, I understand that
at 1.8 GHz all numbers will decrease accordingly.
If I calculate above correctly, then:

Can I get close to 40.8 GB / s in a simple MPI PUT to another node -
or will the limit be 6.8 GB / s?
Then HDR will give more bandwidth (in Ookami w/Infiniband: 19.4 GB / s
as maximum for OSU MPI).
Ookami works w/Infiniband, not TofuD because of such bandwidth for a
not very large cluster - or because of financial reasons (cost of
TofuD routers?) ?

Mikhail Kuzminsky