[Beowulf] how fast can NFS run?

[Bruce Allen]

Hi Joe,

>> don't understand or believe this.  If the server's local disks can 
>> read/write at 300MB/s and the networking can run substantially faster than 
>> 100 MB/s, I don't see any constraint to faster operation.  But perhaps 
>> someone on this list can provide real-world data (or say why it can't 
>> work).
>
> .... ok, a number of different issues going on here
>
> a) the 300 MB/s (SATA II, right?) is the max theoretical speed.  You are 
> going to get something close to this in pure buffer to memory transactions in 
> specialized cases. Normally you will see 50-70 MB/s for these disks for large 
> block sequential reads.  SATA also does a bit of interrupting... you need a 
> *good* SATA controller, or you will see your interrupt rate go up 10x in 
> heavy disk load times.  Software RAID will increase this a bit as well.

The NFS server is using RAID controllers, in PCI-X busses.  Many disks are 
striped together.  We've measured (locally) > 350 MB/s read and > 250 
MB/sec write on these arrays.

> b) If this is gigabit, you get about 110 MB/s max in best case scenarios, 
> with the wind at your packets, along with a nice gravitational potential, an 
> a good switch to direct packets by.  If this is IB, you should be able to see 
> quite a bit higher, though your PCI is going to limit you.  PCI-e is better 
> (and HTX is *awesome*).

This is either 10Gb/s or three or four channel-bonded Gb/s links.  This 
should give substantially faster than 100MB/sec.

Note that even PCI-X is 133 MHz x 8 bytes = 1064 MB/sec.

> FWIW, we have customers with units we have built out that happily support 
> 2-400 MB/s over NFS without complaining, over gigabit (multiple simultaneous 
> clients hammering on the server).  There are multiple problems to overcome to 
> get this working correctly and efficiently.

I'd be interested to hear about the details of this server, on list or off 
list.

Cheers,
 	Bruce