[Beowulf] Small files

[Christopher Samuel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/06/14 05:03, Tom Harvill wrote:

> We've found that a large majority of our files (~40MM of ~50MM)
> are less than 10KB. We believe our filesystem (lustre) is
> bottlenecked with IOPs and locking related to jobs running against
> these files. We have ~700TB usable storage with ~500TB consumed,
> almost all consumption is by a relatively small number of very very
> large files.

Coincidentally I was talking to some people involved in Lustre
yesterday about it and I asked them how well it coped.  There was a
bit of shuffling of feet and the answer was along the lines that it
wasn't really their target use case.

Our GPFS filesystem has a lot of fairly small files, but nothing like
the proportion that you are seeing (4/5ths < 10kB).

We have had odd edge cases, like people running OpenFOAM creating a
directory hierarchy of millions of directories, each with 5 files in
them.   That was fun...

All the best!
Chris
- -- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlOanGIACgkQO2KABBYQAh/ZSQCcCsHfUwWLt5H4mPBij+IAqQsz
XVcAnAtFPoLVF53qymI8KwieqGo7cQMl
=590L
-----END PGP SIGNATURE-----