[Beowulf] Clearing out scratch space

[Ellis H. Wilson III]

On 06/12/2018 04:06 AM, John Hearns via Beowulf wrote:
> In the topic on avoiding fragmentation Chris Samuel wrote:
> 
>  >Our trick in Slurm is to use the slurmdprolog script to set an XFS project
>  >quota for that job ID on the per-job directory (created by a plugin which
>  >also makes subdirectories there that it maps to /tmp and /var/tmp for the
>  >job) on the XFS partition used for local scratch on the node.
> 
> I had never thought of that, and it is a very neat thing to do.
> What I would like to discuss is the more general topic of clearing files 
> from 'fast' storage.
> Many sites I have seen have dedicated fast/parallel storage which is 
> referred to as scratch space.
> The intention is to use this scratch space for the duration of a 
> project, as it is expensive.
> However I have often seen that the scratch space i used as permanent 
> storage, contrary to the intentions of whoever sized it, paid for it and 
> installed it.
> 
> I feel that the simplistic 'run a cron job and delete files older than N 
> days' is outdated.
> 
> My personal take is that heirarchical storage is the answere, 
> automatically pushing files to slower and cheaper tiers.

Disclaimer: I work for one such parallel and fast (and often used for 
scratch) company called Panasas.

I disagree with the notion that hierarchical storage is a silver bullet 
in many HPC-oriented cases.  In fact, I would argue in some environments 
it poses serious risks to being able to keep a lid on your storage cost 
and DC footprint, whether that's for scratch, home, or archive storage. 
People (including myself in my own systems testing) can generate an 
enormous amount of data that has near-zero value past the project 
currently being worked on, which may only be measured in weeks or low 
numbers of months.  In many of my cases it's cheaper to regenerate those 
many TBs of data then to hold onto it for a year or more.  Auto-tiering 
scratch data to cheaper storage as it gets colder seems like an easy 
answer as it takes some of this responsibility away from the users, but 
you'll still want to /someday/ ditch that data entirely (for 
scratch-like data that is).  Culling through piles of likely 
mechanically named files you haven't looked at in a long time is 
difficult as a human exercise, and without sufficiently complex media 
asset management it's also difficult from a storage perspective as your 
data may take a /long/ time to even list, much less grep through, when 
pulling from true archive storage.

For true scratch I think the solution presented by many of the posters 
relating to automatic deletion policies managed by administrators, which 
develops and forces good data habits, is ultimately the cleanest 
solution in the long-term.

Now, tiering within a storage layer based on different types or access 
frequencies of data is perfectly reasonable and is something we do in 
our systems.  Also, using external software to automatically tier cold 
but persistent data (i.e., home dir data) from fast to archive storage 
is also reasonable.  But there are a lot of pitfalls from trying to 
automatically tier data that isn't supposed to be (eternally) persistent 
IMHO.

Best,

ellis

-- 
Ellis H. Wilson III, Ph.D.
      www.ellisv3.com