[Beowulf] Clearing out scratch space

[Ellis H. Wilson III]

On 06/12/2018 10:25 AM, John Hearns via Beowulf wrote:
>  > Disclaimer: I work for one such parallel and fast (and often used for 
> scratch) company called Panasas.
> Ellis, I know Panasas well of course.  You are a great bunch of guys and 
> girls, and have pulled my chestnuts from the fire many times (such as 
> the plaintive call from the customer - we can't access our data. What 
> are all these red lights for - wwe have seen them for weeks. Me - puts 
> in call to 800-PANASAS immediately)
> Having been there on my hands and knees installing Panasas systems for 
> UK Government defence customers.
> Installed Panasas in Formula 1 where it had a dramatic effect on our 
> solver times.
> Also installing the largest Panasas setup in the world at the UK 
> Rutherford Appleton Laboratory, which is being used for lcimate resarch 
> (JASMIN).
> Been there, seen it, unboxed hundreds of blades.

Very glad you've had such good experiences John.  As we like to say 
internally, people buy for the filesystem engineering at Panasas, but 
stay for the service, and that's still probably giving our architecture 
and engineering teams too much credit.  Our support team puts the rest 
of us to shame.

Returning to the subject at hand, I'm wondering aloud if people prefer 
automatic deletion policies for scratch volumes to be incorporated into 
the filesystem or not.  I ask because while external software is more 
flexible in that it should run fine on any POSIX-compliant filesystem, 
it has no good way of assuring thousands of deletions per second when a 
batch of files age-out aren't interfering with foreground performance. 
If instead such aging policies for a scratch volume were more tightly 
integrated into the filesystem in question it could balance issuing the 
deletes with maintaining a QoS level for foreground performance.

We don't have such a feature, but we have much of the infrastructure to 
support a specialized volume type like that without a huge lift.  If 
people were sufficiently interested in it I could definitely see us 
dedicating some engineering resources to it.

Best,

ellis

-- 
Ellis H. Wilson III, Ph.D.
      www.ellisv3.com