[Beowulf] dedupe filesystem
kilian.cavalotti.work at gmail.com
Fri Jun 5 07:09:54 PDT 2009
On Friday 05 June 2009 15:52:55 Mark Hahn wrote:
> > The best of both worlds would certainly be a central, fast storage
> > filesystem, coupled with a hierarchical storage management system.
> I'm not sure - is there some clear indication that one level of storage is
> not good enough?
I guess it strongly depends on your workload and applications. If your users
tend to keep all their files for long-term purposes, as Bogdan Costescu
pertinently described earlier, it might make sense to transparently free up
the fast centralized filesystem and move the unused-at-the-moment-but-still-
crucially-important files to a slower, farther filesystem (or tapes).
This way, you have more fast storage space available for running jobs, while
keeping the convenience for users to still be able to access their archived
files transparently, as if they still were on the filesystem.
It's a nice feature to have because it makes users life easier. Obviously, if
you don't already have this kind of second level storage infrastructure, the
benefit is maybe not worth the investment.
> this seems like a bad design to me. I would think (and I'm reasonably
> familiar with Lustre, though not an internals expert) that if you're going
> to touch Lustre interfaces at all, you should simply add cheaper,
> higher-density OSTs, and make more intelligent placement/migration
In Lustre, that would be done through OST pools. Eh, isn't this also a feature
CEA contributed to? :)
More information about the Beowulf