[Beowulf] bandwidth: who needs it?

[Greg Lindahl]

On Thu, Oct 21, 2004 at 11:53:17AM -0700, Jim Lux wrote:

> I'm sure that there are
> high performance switches around that only wait 'til enough of the header
> arrives to make the routing decision, but then, the switch has to passively
> pass the data through without error checking.

Actually, it's more common to route immediately after you've seen the
header, but to compute the entire CRC and then do something minimal if
the CRC turns out to be wrong.  I may be confusing the exact details,
but I think that Infiniband just counts the bad packets and depends on
the endpoint to discard the packet. Myrinet both counts the error, and
sticks a zero into the CRC, so that subsequent hops will know that the
CRC was found to be bad earlier in the path.

GigE and 10 gigE switches receive the whole packet before sending it
on, and so the higher bandwidth is a huge help for 10 gig's latency --
drops it from 15 usec for a 1500 byte packet to 1.5 usec.

-- greg