[Beowulf] Small files
Jason Riedy
jason at lovesgoodfood.com
Fri Jun 13 06:53:09 PDT 2014
And Skylar Thompson writes:
> There seems to be a trend in the bioinformatics community for
> using the filesystem as a database.
Well, the file system *is* a hierarchical key-value store, and
it's one that just about every program can use without special
adapters, syntax, and/or complexity. Users already are familiar
with tons of tools for querying and manipulating the file system,
etc. And it's stupidly easy to parallelize over keys (files),
only re-run jobs that have changed, etc. using common tools.
I know there's some work out there on single-node file systems
layered on fast k/v stores. I wonder if it wouldn't be wiser to
build a FS on a distributed k/v store for these uses...
More information about the Beowulf
mailing list