[Beowulf] Storage Best Practices

[Adam DeConinck]

The best success I've seen is a mix of strategies: some amount storage
that's considered "permanent" with aggressive quota limits (per user
or per project), plus a larger no-quota space that's cleaned up based
on some retention policy. I've used this basic scheme in small
environments, and also seen it used successfully in large institutions
(where I was not directly responsible for storage).

That said, there's no way to make everyone happy with this -- people
will complain when a file is deleted unexpectedly, or when they hit
their quota, or both. Time-based retention policies that work by
automatically checking file modification time also invite users to
write "touch" scripts to try to fool the cleaner. It can turn into a
fun game. But I've also used these kinds of policies to justify new
storage purchases, and make users into allies in funding them.

On a lighter note, this is a good excuse for me to link my favorite
James Cuff post ;-) -
http://blog.jcuff.net/2011/12/cautionary-tale-about-storage-and.html

On Mon, Feb 19, 2018 at 6:12 AM, Richter, Brian J {BIS}
<Brian.J.Richter at pepsico.com> wrote:
> Hey All,
>
>
>
> I was hoping to get some recommendations for Storage. Last year we set up
> our first HPC and I’m looking for a good strategy moving forward for
> Storage. We set up a dedicated space on the cluster for Storage that has 5.5
> TB of space. This space can be quickly chewed up depending on the project
> the business is working on. Do you guys typically do a retention policy on
> the cluster to ensure there is always enough space? I was thinking anything
> older than a month should be cleaned up.
>
>
>
> Thanks!
>
>
>
> Brian J. Richter
>
> Global R&D Senior Analyst • Information Technology
>
> 617 W Main St, Barrington, IL 60010
> Office: 847-304-2356 • Mobile: 847-305-6306
>
> Brian.J.Richter at pepsico.com
>
>
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>