[Beowulf] diskless booting over jumbo frames

[Amrik Singh]

I am sorry for not making things more clear. scp was faster then NFS 
only when NFS server was being hammered by the client nodes. This 
happens only when number of client nodes increase above a certain 
number. In my test I was reading files (1-2 MB) from an NFS server from 
180 nodes. All the nodes were reading different files from different 
folders and there was no caching. There was a huge io wait time on the 
server CPUs.

The setup is like this:

Server exports /data
Clients mount /data and read files from /data/node1, /data/node2 and so on.

While the server is busy, if I try to do

node1:/tmp# scp largefile1 server:/data/node1
node1:/tmp# cp largefile2 /data/node1

Now in this situation scp is faster then the cp. This lead me to believe 
that NFS is the bottleneck when it hammered by requests from too many nodes.


Amrik 



Bogdan Costescu wrote:
> On Wed, 25 Apr 2007, Amrik Singh wrote:
>
>> I agree that Jumbo Frames would not be a great help with the root 
>> file system but we hope to get a better performance from other NFS 
>> servers.
>
> I don't quite follow you here: the link that was sent earlier had 
> shown some kernel level autoconfig problem. If this is indeed still 
> the case with newer kernels, this should only affect the root FS - 
> before mounting the other NFS exports, you should have the chance to 
> perform a proper initialization of the network, which would give you 
> the possibility to use a larger MTU.
>
>> As all the machines on the same subnet have to be using the jumbo 
>> frames,
>
> Why ? The MTU specifies a _maximum_ value; even when using 1500, not 
> all packets are exactly 1500 bytes. The larger MTU _allows_ larger 
> values, but doesn't _force_ them.
>
>> Even though NFS is extremely slow, copying files over scp is still 
>> very fast between a client and server.
>
> If by this you mean that NFS is slower than scp, then this should be 
> exactly the other way around, because scp needs CPU time for its 
> encryption. I would have said that you have some network problems and 
> use NFS over UDP which leads to high retransmission rates; scp would 
> adapt better to the network problems due to using TCP... but you just 
> mention that you tried switching between TCP and UDP.
>
>> We have tried all different ways to tune the NFS for a better 
>> performance (increasing NFS deamons on the servers, changing rsize & 
>> wsize, using TCP vs UDP, using async vs sync, noatime, timeo).
>
> Could it be that you tried so hard to tune it that you just used too 
> many settings that don't play well together in your particular 
> situation ? I've found the NFS client and server in the kernels of the 
> recent years to perform reasonably well in their default 
> configuration; although they could be further optimized, a NFS 
> transfer in these conditions would always beat a scp one.
>
>> initrd=bzImage ramdisk=40960
>
> dhclient could be located in the ramdisk. Actually even a whole root 
> FS could be located in the ramdisk, if there are not too many nodes 
> booting at the same time and leading to UDP packet loss.
>
> ... and you already have a rather large ramdisk. Have you created it 
> yourself (and therefore know what's in it and how to add some more) ?
>