[Beowulf] Slightly OT: storage performance

[Buccaneer for Hire.]

----- Original Message ----
From: Joe Landman <landman at scalableinformatics.com>

[snip]

>   My question is this:  apart from using huge file sizes to see raw 
> disk performance, what do you considered good performance on the various 
> tests, either in the huge file size regime, or in the cache interaction 
> regime?  Basically which tests are most meaningful to your workloads? 
> Are the raw disk data really the most useful datum?  Are they corner 
> cases that you are simply interested in?  Is the most important test 
> case reading and changing one byte at random in a 1TB file, several 
> hundred million times?  Or is it large block sequential IO?

The good thing about using IOZone and Bonnie++ is that there is a large body of numbers that you can compare your equipment to.  We also have a program written in-house that we have testing and verified over and over again.  Besides the normal sequential read/write and single/multithread randoms, it can also mimic how seismic data is process in our real world.

The minimum file size for testing should be at least twice the maximum
estimated cache in the system.  So, we use the amount of memory on the
head node and all the cache on the EMC and double it.  Then you start test and tune the file sytems and local vm system, then you tune the networking part of your system.  Then you start the remote testing.  We will throw 60-100 nodes at it blasting a full speed, each building the same file size so cache is irrelevant and see how the raw numbers are (aggregate bandswith on the remote end) and how the array is handling.  We can currently do 180MB peak over NFS with the array pushing  210+MB using a CX700.

After a while testing, you get an idea for what your workload requires.  You we have a rule of thumb for our environment (though it may change as I test the new mode of operation discussed before.)   We have three basic criteria-reliability, speed (though we are willing to give up a little speed for reliability), and capacity (though we are willing to give up a little capacity for speed).  And of course we try to get the most we can get for the money with above in mind. So, our current purchase is 112TB useable of EMC space.

> If there is some sort of cutoff that people have between what they 
> consider "eh" and "good", I would like to hear it.  You can email me 
> offline if you want, and I will summarize later on.

Corporate decisions may (and has) modify what we get...

HTH
-ARR






 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index