[Beowulf] Help for terrible NFS write performance

[Cunningham, Dave]

Are you suggesting that running nosync in an hpc environment might be a good idea ?  We had that discussion with our integrator and were assured that running nosync was the path to damnation and would probably grow hair on our palms.

What are your thoughts on the tradeoff ?

   Dave Cunningham

-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Joe Landman
Sent: Friday, August 21, 2009 10:38 AM
To: Orion Poplawski
Cc: Beowulf List
Subject: Re: [Beowulf] Help for terrible NFS write performance

Orion Poplawski wrote:
> I'm trying to improve the terrible NFS (write in particular) performance 
> I'm seeing.  Pure network performance does not appear to be an issue as 
> I can hit 120MB/s reading which should be about the limit for gigE. 
> Perhaps the local disk performance is not what it should be.  Any help 
> would be greatly appreciated.  Using bonnie++ for benchmarks.
> 
> Server:
> 
> Dual proc dual core opteron 2GHz
> 8GB RAM
> CentOS 4.7
> kernel  2.6.9-78.0.22.plus.c4smp
> 3 8-port Marvell MV88SX6081 SATAII controllers
> sata_mv 3.6.2 driver
> Ethernet controller: nVidia Corporation MCP55 Ethernet (rev a3)
> MTU 8982
> 
> Arrays are linux md arrays of 6 disks with 2 on each controller.  64k 
> cunks.  ext3 filesystem.

If I had to bet, ext3 would have much to do with this ... though, 
honestly, md RAID write performance over NFS is nothing to write home 
about.  We can get ~350MB/s on our DeltaV's, but this takes lots of work.

> 
> "working" - raid0 ST31000340AS 1TB drives
> local perf: 224-240MB/s write, 135MB/s rewrite, 390-400MB/s read
> "cora6" - raid5  ST31500341AS 1.5TB drives
> local perf: 84MB/s write, 42MB/s rewrite, 161-166MB/s read
> 
> /etc/exports:
> /export          *.cora.nwra.com(rw,sync,fsid=0)
> /export/cora6    *.cora.nwra.com(rw,sync,nohide)
> /export/working  *.cora.nwra.com(rw,sync,nohide)

Ok.  There it is...  Sync.

Don't need to see anything else.

That is it.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf