[Beowulf] Small files

Jason Riedy jason at lovesgoodfood.com
Fri Jun 13 06:53:09 PDT 2014


And Skylar Thompson writes:
> There seems to be a trend in the bioinformatics community for
> using the filesystem as a database.

Well, the file system *is* a hierarchical key-value store, and
it's one that just about every program can use without special
adapters, syntax, and/or complexity.  Users already are familiar
with tons of tools for querying and manipulating the file system,
etc.  And it's stupidly easy to parallelize over keys (files),
only re-run jobs that have changed, etc. using common tools.

I know there's some work out there on single-node file systems
layered on fast k/v stores.  I wonder if it wouldn't be wiser to
build a FS on a distributed k/v store for these uses...



More information about the Beowulf mailing list