[Beowulf] Small files

[Brian Dobbins]

Just a quick addition to this thread (for now), here's the presentation I
think Killian refers to:
http://www.opensfs.org/wp-content/uploads/2014/04/D2_S16_PLFSandLustrePerformanceComparison.pdf

And here's *all* the slides from LUG 2014:
http://opensfs.org/lug-2014-presos/

Some of them are incredibly informative, and I have high hopes for some of
the future changes referenced in Andreas Dilger's presentation (
http://opensfs.org/wp-content/uploads/2014/04/D1_S9_LustreFutureFeatures.pdf
) - specifically, I think the combination of DNE (phase 1 & 2) and small
files on MDTs should help immensely with both IOPS and overall throughput
on those sorts of small-file workloads.

Cheers,
  - Brian



On Fri, Jun 13, 2014 at 2:02 PM, Kilian Cavalotti <
kilian.cavalotti.work at gmail.com> wrote:

> On Fri, Jun 13, 2014 at 7:03 AM, Ellis H. Wilson III <ellis at cse.psu.edu>
> wrote:
> > a) Fix it transparently with automatic policies/FS's in the back-end. (I
> > know of at least one FS that packs small files with metadata
> transparently
> > on SSDs to expedite small file IOPS, but message me off-list for that as
> I
> > start work for that shop soon and don't want to so blatantly advertise).
> > There are limits to how much these policies/FS's can fix though.  Bad I/O
> > will still be Bad I/O after a point.
>
> Speaking of which, there's this project from Los Alamos, PLFS
> (https://github.com/plfs/plfs-core), that tries to aggregate random
> I/O from applications into sequential operations that better suit the
> underlying parallel filesystem. Never used it myself, but I've seen a
> presentation from Brett Kettering at LUG'14 which outlined nice
> performance improvements for some use cases.
>
> Cheers,
> --
> Kilian
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20140613/738e2d55/attachment.html>