[Beowulf] how fast can NFS run?

Bruce Allen ballen at gravity.phys.uwm.edu
Tue Jan 31 21:06:41 PST 2006


Hi Joe,

>> don't understand or believe this.  If the server's local disks can 
>> read/write at 300MB/s and the networking can run substantially faster than 
>> 100 MB/s, I don't see any constraint to faster operation.  But perhaps 
>> someone on this list can provide real-world data (or say why it can't 
>> work).
>
> .... ok, a number of different issues going on here
>
> a) the 300 MB/s (SATA II, right?) is the max theoretical speed.  You are 
> going to get something close to this in pure buffer to memory transactions in 
> specialized cases. Normally you will see 50-70 MB/s for these disks for large 
> block sequential reads.  SATA also does a bit of interrupting... you need a 
> *good* SATA controller, or you will see your interrupt rate go up 10x in 
> heavy disk load times.  Software RAID will increase this a bit as well.

The NFS server is using RAID controllers, in PCI-X busses.  Many disks are 
striped together.  We've measured (locally) > 350 MB/s read and > 250 
MB/sec write on these arrays.

> b) If this is gigabit, you get about 110 MB/s max in best case scenarios, 
> with the wind at your packets, along with a nice gravitational potential, an 
> a good switch to direct packets by.  If this is IB, you should be able to see 
> quite a bit higher, though your PCI is going to limit you.  PCI-e is better 
> (and HTX is *awesome*).

This is either 10Gb/s or three or four channel-bonded Gb/s links.  This 
should give substantially faster than 100MB/sec.

Note that even PCI-X is 133 MHz x 8 bytes = 1064 MB/sec.

> FWIW, we have customers with units we have built out that happily support 
> 2-400 MB/s over NFS without complaining, over gigabit (multiple simultaneous 
> clients hammering on the server).  There are multiple problems to overcome to 
> get this working correctly and efficiently.

I'd be interested to hear about the details of this server, on list or off 
list.

Cheers,
 	Bruce



More information about the Beowulf mailing list