[Beowulf] GPFS on Linux (x86)

[Brent Franks]

On 9/14/06, Scott Atchley <atchley at myri.com> wrote:
> You do not mention what interconnect (protocol or advertised
> throughput) that you are considering. If you are considering gigabit
> Ethernet, you can skip the rest of this message.

All,

Thank you very much for your input, comments, contacts, and
suggestions.  This information is greatly appreciated and will go a
long way in helping us in overcoming the performance issues we
currently face.

We are currently running GigE as our interconnect.

Basically we are currently running two NFS servers out to our web
servers.  We also are running three MySQL servers. The MySQL instances
are segmented right now, but we are about to start an eval of
Continuent's M//Cluster software.

As stated, our FS infrasdtructure leaves much to be desired.  The
current setup involving NFS servers (Dell PE 2850 with local 1TB local
storage 10K scsi disks) have not performed well.  We are constantly IO
waiting.

Another interesting thing is, each MySQL server is using a ISCSI block
device from SATAII NAS servers that we built using generic super micro
boards and Areca controllers.  Each of these boxes has approx 2.1TB of
usable disk, and the performance has been suprisingly good.  The Areca
1160 controllers with 1GB cache are handling the load, especially
compared to our FS infrastructure of localized disks (I would have
thought the opposite would be true), as the mysql disk IO pattern
would be more smaller random IO, and the FS is mostly read (serving up
web pages).

We have made pretty much every last ounce of optimization we can on
the NFS side (TCP, packet sizes, hugemem kernels, tried David Howells
fscache on web client side) but non has been the silver bullet we've
been looking for, which led us down the parallel fs path.

We were hoping to ultimately employ something like the following:

Take the ISCSI target boxes, the two nfs servers with local disk, and
possibly purchase additional hardware if necessary, and run a parallel
fs across them.  GPFS looked the most feature packed (concurrent file
access, multiple read/write, client side caching, PFS Data Striping,
and storage pools).

I think ultimately the way to describe our current setup is we out
grown the sweet spot of the ad-hoc NFS, mysql approach we are
currently employing, vs. not being large enough quite yet for the full
on Cluster FS that we seek to employ.. Yet in an effort to scale to
the sky, we are going to try to do this correctly, rather than
continually being reactive.

Best Regards,

- Brent