[Beowulf] how fast can NFS run?
ballen at gravity.phys.uwm.edu
Tue Jan 31 21:06:41 PST 2006
>> don't understand or believe this. If the server's local disks can
>> read/write at 300MB/s and the networking can run substantially faster than
>> 100 MB/s, I don't see any constraint to faster operation. But perhaps
>> someone on this list can provide real-world data (or say why it can't
> .... ok, a number of different issues going on here
> a) the 300 MB/s (SATA II, right?) is the max theoretical speed. You are
> going to get something close to this in pure buffer to memory transactions in
> specialized cases. Normally you will see 50-70 MB/s for these disks for large
> block sequential reads. SATA also does a bit of interrupting... you need a
> *good* SATA controller, or you will see your interrupt rate go up 10x in
> heavy disk load times. Software RAID will increase this a bit as well.
The NFS server is using RAID controllers, in PCI-X busses. Many disks are
striped together. We've measured (locally) > 350 MB/s read and > 250
MB/sec write on these arrays.
> b) If this is gigabit, you get about 110 MB/s max in best case scenarios,
> with the wind at your packets, along with a nice gravitational potential, an
> a good switch to direct packets by. If this is IB, you should be able to see
> quite a bit higher, though your PCI is going to limit you. PCI-e is better
> (and HTX is *awesome*).
This is either 10Gb/s or three or four channel-bonded Gb/s links. This
should give substantially faster than 100MB/sec.
Note that even PCI-X is 133 MHz x 8 bytes = 1064 MB/sec.
> FWIW, we have customers with units we have built out that happily support
> 2-400 MB/s over NFS without complaining, over gigabit (multiple simultaneous
> clients hammering on the server). There are multiple problems to overcome to
> get this working correctly and efficiently.
I'd be interested to hear about the details of this server, on list or off
More information about the Beowulf