[Beowulf] LVM performance problems

[Bill Broadley]

Judd Tracy wrote:
> I am trying to bring up a small file server and am noticing some serious
> performance issues when using LVM.  I created a software raid /dev/md0

How many drives?  Which raid level?  What stripesize?  What RAID controller?

> which I can read at ~195MB/s using the raw raid device, but as soon as I

You did benchmark reads using accesses significantly larger than ram... right?

> put LVM on top of it the read speeds drop to ~95MB/s using a raw lvm

Exactly what config in LVM?  What stripesize?

> partition without a filesystem.  When I use and xfs filesystem on top of
> the lvm partition it drops down to ~40MB/s.  This seems like a

Exactly what mkfs.xfs parameters did you use?  In particular switch, sunit,
and related parameters.

> processing power issue, but the machine is a dual processor opteron
> system with 4GB of ram in it.  Does anyone have some insight into why I
> might be having such problems with the system.

I've not seen many differences in my testing.  Drives are capable of
30-60MB/sec each for sequential reads/writes, the trick is to balance
the load across all disks for your workload.  Beware, optimizing your
setup for dd, and bonnie might lead to worse performance if your workload
doesn't use similar access patterns.

The interaction between N disks, reads/writes of M size, and various strip
sizes (in the raid and in the filesystem) is rather complex.

So my guess is that your access pattern used to hit many of your disks
in parallel, but now is bottlenecked by a single disk.  Not that you are
somehow CPU limited.