[Beowulf] scheduler policy design

[Chris Samuel]

On Thu, 26 Apr 2007, Leif Nixon wrote:

> We do advance reservation with backfill, so short jobs will on average
> start sooner than long jobs.

We use 3 different metrics to work out queue priorities on our clusters:

1. X-Factor - Maui/Moab speak for a ratio of queue time to walltime [1]. 
Increases much faster for short jobs over longer jobs and (as Lief mentions)
encourages users to estimate their jobs if they are shorter than the default
time we give them of 14 days (if they can do this).

2. Number of CPUs - We work on the principle that it is easier to fit smaller
jobs in around larger ones on a system than the other way around. The Tetris
effect.

3.  FairShare - Each of our member Universities has a percentage share
based on their annual subscription (which in turn is calculated from some
research numbers).  Users from institutions who are under quota get a
boost whilst queued and users from over-quota institutions tend to have to
wait longer (we are always over-subscribed).

4. Bypass - any eligible job gets a small boost every time the scheduler
backfills another job in front of it.

On top of that we have hard and soft quotas on the number of jobs a user can
have running and the number of CPUs they have running plus we have a limit on
the number of jobs that they can have eligible to run at any one point.  Jobs
that exceed those policy limits are "blocked" and are not eligible to run and
do not have their priority adjusted until they become eligible.

This stops people queue stuffing with hundreds of jobs and causing other users
to end up waiting an awful long time before resources become free.

As a result our clusters have run at a combined utilisation of 84% over the last
12 months.

cheers!
Chris

[1] - http://www.clusterresources.com/products/mwm/docs/5.1.2priorityfactors.shtml#xfactorsub

-- 
 Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager
 Victorian Partnership for Advanced Computing http://www.vpac.org/
 Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia