[Beowulf] Anyone have docs/URLs/resources covering Grid Engine vs SLURM deltas or migration?

[Alex Chekholko]

Hi Chris,

Having gone through this migration on several small clusters over the
years, I can say that initially it seems pretty scary but in the end it was
all actually very straightforward.  Obviously all the names of the
resources and queues/partitions change and the user command names change
but all the concepts stay the same and map directly.  Much like migrating
from torque to sge back in the day.

For specific technical details, I like this wiki, maintained by one person
from the slurm mailing list:
https://wiki.fysik.dtu.dk/niflheim/SLURM

Regards,
Alex

On Fri, Jan 29, 2021 at 12:13 PM Chris Dagdigian <dag at sonsorol.org> wrote:

> Hi folks,
>
> Those who know me from my day job and other email address know that I've
> been a hardcore SGE person for a decade+ now.  Feels bad to even write
> this email, heh
>
> Internally at my company we've been having a great internal discussion
> about SGE and SLURM, the specific trigger being our widespread use of
> AWS Parallelcluster for auto-scaling compute farms on AWS and the
> decision by AWS to deprecate the open source SGE distributed with the
> stack at some point in the future.
>
> Related to this is the longstanding use of SGE in the life sciences --
> there are genome sequencers and other wet lab instruments that ship
> natively with SGE support from the vendor so there is going to be a long
> tail of SGE still in use or targeted for use in biotech and pharma spaces.
>
> I think there is still a future for commercial SGE especially after the
> Univa -> Altair tie up but that still leaves the poor orphaned/forked
> open source SGE distro's still kinda hanging out there with no real
> updates or improvements in ages so I am understanding of the AWS HPC
> folk desire to pare down their supported scheduler stack.
>
> I want to build up my own knowledge and prep for our own increased use
> of SLURM on AWS.  I'm very comfortable with SGE architecture,
> operational philosophy and capabilities but I lack similar info for
> modern SLURM.   I'm ready and willing to start from scratch to build my
> own transition and "differences between SGE/SLURM" documentation but was
> wondering who out there has made this transition before and if there are
> any public domain FAQs, wikis, technical writeups or other guidance that
> I can learn from.
>
> If I can manage to put my own materials together and they look sensible
> I will plan on publishing them openly. Thanks!
>
> So far the internal conversation we are having is centering on the
> differences in resource based job scheduling when there are specific
> needs to declare required resources up front like GPUs or memory
> requirements.  Most of the differences beyond basic queue/partition
> design seem to center around the minutiae of scheduling and placing jobs
> but I'm sure I'm missing other larger areas.
>
>
> Regards
> Chris
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20210129/61c4ffbb/attachment.htm>