[Beowulf] MPI-IO + nfs - alternatives?

[Shai Fultheim (Shai@ScaleMP.com)]

Robert,

I would go for virtualizing the cluster as single system (virtual SMP) and then using local I/O - just as we all run MPI apps on large SMPs years ago.
Specifically, vSMP Foundation (www.scalemp.com) provides great scratch performance with local drives (use Linux raid utilities to make RAID0).

Best regards,
shai

--Shai

 Visit us at SC10, Nov 13-19,  New Orleans, booth 3239


-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Robert Horton
Sent: Wednesday, September 29, 2010 18:24
To: Beowulf Mailing List
Subject: [Beowulf] MPI-IO + nfs - alternatives?

Hi,

I've been running some benchmarks on a new fileserver which we are
intending to use to serve scratch space via nfs. In order to support
MPI-IO I need to mount with the "noac" option. Unfortunately this takes
the write block performance from around 100 to 20MB/s which is a bit
annoying given that most of the workload isn't MPI-IO.

1) Does anyone have any hints for improving the nfs performance under
these circumstances? I've tried using jumbo frames, different
filesystems, having the log device on an SSD and increasing the nfs
block size to 1MB, none of which have any significant effect.

2) Are there any reasonable alternatives to nfs in this situation? The
main possibilities seem to be:

 - PVFS or similar with a single IO server. Not sure what performance I
should expect from this though, and it's a lot more complex than nfs.

 - Sharing a block device via iSCSI and using GFS, although this is also
going to be somewhat complex and I can't find any evidence that MPI-IO
will even work with GFS.

Otherwise it looks though the best bet would be to export two volumes
via nfs, only one of which is mounted with noac. Any other suggestions?

Rob

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf