[Beowulf] OT: effective amount of data through gigabit ether?

[Robert G. Brown]

On Mon, 4 Oct 2004, Michael Will wrote:

> On Monday 04 October 2004 01:40 pm, Mike wrote:
> > I know this is off topic, but I've not found an answer anywhere.
> > On one IBM doc it says the effective throughput for 10Mb/s is
> > 5.7GB/hour, 100Mb/s is 17.6GB/hour, but only lists TBD for 1000MB/s.
> 
> I would assume 900Mb/s as an optimistic best case throughput for the GigE, 
> which would be about 395GB/hour.  
> 
> 17.6GB/hour seems like a really low estimate, that would be only
> about 40Mb/s effective transfer rate over an 100Mb/s link? Maybe
> that number is really measuring the tape writing speed instead?


I agree with Michael (and Sean) here -- it is pretty straightforward to
compute a theoretical peak bandwidth -- just convert the Mbps into MBps
by dividing by 8.  10 Mbps ethernet can thus manage 1.25 MBps, 100 Mbps
-> 12.5 MBps, 1000 Mbps -> 125 MBps.

Most people don't consider the packet headers to be part of the
"throughput" -- with a standard ethernet MTU of 1500 bytes + 18 bytes
ethernet header, take away 64 bytes for TCP/IP header, one has at most
1436/1518 or 94.6% of peak This leaves a STILL theoretical peak of 1.18
x 10^{n-1} MBps (where n is the log_10 of the raw BW in Mbps).  One
reason people like to use switches and NICs that support oversize
packets is that doing so reduces both this 5.4% chunk of header-based
overhead and the ordinarily "invisible" mandatory pause between packets,
per packet, letting you get a bit closer to peak.

On top of this, switches and so forth will typically add a small bit of
latency per packet on top of the minimum interpacket interval, and the
TCP stack on both ends of the connection will add another slice of
latency per packet.  These are typically of order 50-200 microseconds
(which end of this wide range you see depending on lots of things like
switch quality and load, NIC quality and type, CPU speed and load, OS
revision, and probably whether or not it is a Tuesday).  By the time all
is said and done, one usually ends up with somewhere between 80% and 94%
of theoretical peak, or 1.0 x 10^{n-1| and 1.174 x 10&{n-1} MBps,
although for particularly poor NICs I've done worse than 80% in years
past.  Higher numbers (closer to theoretical peak, as noted) for giant
packets.

This is likely to be the relevant estimate for moving large data files.
For moving SMALL data files or messages, one moves from
bandwidth-dominated communications to latency-dominated communications
bottlenecks.  For small messages (say, less than 1K for the sake of
argument) the "bandwidth" increasingly is simply the size of the data
portion of the packet times the number of packets per second your
interface can manage.

To convert into GBph is trivial:  there are 3600 seconds/hour, and 1000
MB in on GB, so multiplying the numbers above by 3.6 seems in order.
This gives a theoretical peak in the ballpark of 4.25 x 10^{n - 1} GBph
for a standard MTU (higher with large packets), a probable real world
peak more like 4.05 x 10^{n - 1} GBph at 90% of wirespeed.

FWIW, I think that GbE tends to perform closer to its theoretical peak
than do the older 10 or 100 BT.  This is both because it is much more
likely that the interfaces and switches will handle large frames and
because the hardware tends to be more expensive and better built, with
more attention paid to details like how things are cached and DMA that
can make a big difference in overall performance efficiency.

Hope this helps, although as has already been noted (and will likely be
noted again:-) the network isn't necessarily going to be the rate
limiting bottleneck for backup.

   rgb

> 
> Michael Will
> > Does anyone know what this effective number is? This is for
> > calculating how long backups should take through my backup network.
> > 
> > (I'm not interested in how long it takes to read/write the disk,
> > just the network throughput.)
> > 
> > Mike
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> > 
> 
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu