[Beowulf] Big storage

[Lombard, David N]

On Tue, Sep 11, 2007 at 11:54:01AM -0400, Joe Landman wrote:
> Loic Tortay wrote:
> 
> > My pointless result was of course mostly due to cache, with 4 threads
> > each writing 1 Gbyte to 4 existing 2 GBytes files (one file per
> > thread).  The block size used was 128 kBytes, all (random) accesses are
> > block aligned, the value is the average aggregated throughput of all
> > threads for a 20 minutes run.
> 
> I seem to remember being told in a matter of fact manner by someone some
> time ago, that only 2GB of IO mattered to them (which was entirely
> cached BTW), so thats how they measured.  Caused me some head
> scratching, but, well, ok.

If that was *truly* their usage, it *is* ok.  But...

> My (large) concern on iozone and related is that it spends most of its
> time *in cache*.  Its funny, if you go look at the disks during the
> smaller tests, the blinkenlights don't blinken all that often ...
> (certainly not below 2GB or so).

Agreed--it all depends on the phys memory; failing to consider that
can lead to invalid results.

> Then again, maybe IOzone should be renamed "cache-zone" :)  More
> seriously, I made some quick source changes to be able to do IOzone far
> outside cache sizes (and main memory sizes) so I could see what impact
> this has on the system.  It does have a noticable impact, and I report
> on it in the benchmark report.

I've found specifying the appropriate file size works well.  For a while
now, filesize > 2*memorysize has gotten to uncached performance for
directly connected devices.  Complications can clearly occur when
there's additional caching on the other end of the wire.

What have you needed beyond the various sync options, e.g., -e, -G, -k,
-o, -r?

-- 
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.