[Beowulf] slurm in heterogenous cluster

[Christopher Samuel]

Hi Mikhail,

On 18/09/17 15:41, Mikhail Kuzminsky wrote:

> Is it possible to use diffenent slurm versions on different worker nodes
> of cluster (w/other slurmctld and slurmdbd versions on head node) ? If
> this is possible in principle (to use different slurmd versions on
> different worker nodes), what are the most important restrictions for this?

The best info is in the "Upgrading" section of the Slurm quickstart guide:

https://slurm.schedmd.com/quickstart_admin.html

# Slurm daemons will support RPCs and state files from the two
# previous minor releases (e.g. a version 16.05.x SlurmDBD will
# support slurmctld daemons and commands with a version of 16.05.x,
# 15.08.x or 14.11.x). [...]
#
# If the SlurmDBD daemon is used, it must be at the same or higher
# minor release number as the Slurmctld daemons. In other words,
# when changing the version to a higher release number (e.g from
# 16.05.x to 17.02.x) always upgrade the SlurmDBD daemon first.
# [...]
#
# The slurmctld daemon must also be upgraded before or at the same
# time as the slurmd daemons on the compute nodes. Generally,
# upgrading Slurm on all of the login and compute nodes is
# recommended, although rolling upgrades are also possible
# (i.e. upgrading the head node(s) first then upgrading the
# compute and login nodes later at various times). Also see
# the note above about reverse compatibility. [...]

I don't know how well you'd go with differing versions across compute
nodes, I'd suggest if you are going to do that you have a partition per
version as I would guess an older version will not like talking to a new
version.

So basically you could have (please double check this!):

slurmdbd: 17.02.x
slurmctld: 17.02.x
slurmd: 17.02.x & 16.05.x & 15.08.x

or:

slurmdbd: 17.02.x
slurmctld: 16.05.x
slurmd: 16.05.x & 15.08.x

or:

slurmdbd: 16.05.x
slurmctld: 16.05.x
slurmd: 16.05.x & 15.08.x

or:

slurmdbd: 16.05.x
slurmctld: 15.08.x
slurmd: 15.08.x

or:

slurmdbd: 15.08.x
slurmctld: 15.08.x
slurmd: 15.08.x


Good luck!
Chris
-- 
 Christopher Samuel        Senior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545