[Beowulf] dedupe filesystem

[Kilian CAVALOTTI]

On Friday 05 June 2009 15:52:55 Mark Hahn wrote:
> > The best of both worlds would certainly be a central, fast storage
> > filesystem, coupled with a hierarchical storage management system.
>
> I'm not sure - is there some clear indication that one level of storage is
> not good enough?

I guess it strongly depends on your workload and applications. If your users 
tend to keep all their files for long-term purposes, as Bogdan Costescu 
pertinently described earlier, it might make sense to transparently free up 
the fast centralized filesystem and move the unused-at-the-moment-but-still-
crucially-important files to a slower, farther filesystem (or tapes). 
This way, you have more fast storage space available for running jobs, while 
keeping the convenience for users to still be able to access their archived 
files transparently, as if they still were on the filesystem. 

It's a nice feature to have because it makes users life easier. Obviously, if 
you don't already have this kind of second level storage infrastructure, the 
benefit is maybe not worth the investment.

> this seems like a bad design to me.  I would think (and I'm reasonably
> familiar with Lustre, though not an internals expert) that if you're going
> to touch Lustre interfaces at all, you should simply add cheaper,
> higher-density OSTs, and make more intelligent placement/migration
> heuristics. 

In Lustre, that would be done through OST pools. Eh, isn't this also a feature 
CEA contributed to? :)

Cheers,
-- 
Kilian