[Beowulf] NFS over RDMA performance confusion

[holway at th.physik.uni-frankfurt.de]

Hi,

I am a bit confused.

I have 4 top notch dual socket machines with 128GB ram each. I also have a
Nexenta box which is my NFS / ZFS server. Everything is connected together
with QDR infiniband.

I want to use this for setup for mysql databases so I am testing for 16K
random and stride performance.

If I set up a single machine to hammer the fileserver with IOzone I see
something like 50,000 IOPS but if all four machines are hammering the
filesystem concurrently we got it up to 180,000 IOPS.

Can anyone tell me what might be the bottleneck on the single machines?
Why can I not get 180,000 IOPS when running on a single machine.

If I test using IPoIB in connected mode I see this: http://pastie.org/4708542

Some kind of buffer problem?

Thanks,

Andrew