[Beowulf] how fast can NFS run?

[Joe Landman]

Bruce Allen wrote:

>> b) If this is gigabit, you get about 110 MB/s max in best case 
>> scenarios, with the wind at your packets, along with a nice 
>> gravitational potential, an a good switch to direct packets by.  If 
>> this is IB, you should be able to see quite a bit higher, though your 
>> PCI is going to limit you.  PCI-e is better (and HTX is *awesome*).
> 
> 
> This is either 10Gb/s or three or four channel-bonded Gb/s links.  This 
> should give substantially faster than 100MB/sec.
> 
> Note that even PCI-X is 133 MHz x 8 bytes = 1064 MB/sec.

Most folks don't see more than 850 MB/s due to overhead and other 
issues.  This is one of the reasons why PCIe is so interesting/useful.

The idea is that you shouldn't be filling this link up.

A simple test between 2 systems like this would be whether or not you 
can push the data over something as simple as a socket (not even NFS) to 
a receiver at the "wire" speed through the switch.  I assume you could 
read it off the disk and shove it out the wire (might even want to just 
stream from disk out the wire, and avoid buffering as this looks like it 
will get in the way of the disk for this scenario).  You could even have 
4 single wire systems to which you open one socket each, and just toss 
data.  c.f. http://netcat.sourceforge.net/

> 
>> FWIW, we have customers with units we have built out that happily 
>> support 2-400 MB/s over NFS without complaining, over gigabit 
>> (multiple simultaneous clients hammering on the server).  There are 
>> multiple problems to overcome to get this working correctly and 
>> efficiently.
> 
> 
> I'd be interested to hear about the details of this server, on list or 
> off list.

ok.

> 
> Cheers,
>     Bruce

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615