[Beowulf] Clearing out scratch space

[Chris Samuel]

On Tuesday, 12 June 2018 6:06:06 PM AEST John Hearns via Beowulf wrote:

> What do most sites do for scratch space?

At ${JOB-1} we used GPFS and so for the scratch filesystem we used the GPFS 
policy engine to identify and remove files that had not been read/written for 
more than the defined number of days (twice the length of time of our longest 
permitted job, so 2 x 30 days).

With the policy engine you can parallelise that across NSD servers too (we had 
separate metadata and data NSD servers, so identifying the files was just 
parallelised across the former which accessed the same shared SSD array). 

Our compute nodes were diskless too so for jobs we used a plugin which created 
per-job directories under scratch and then the epilog would clean them up.

cheers!
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC