[Beowulf] SGE + policy

[Gerry Creager N5JXS]

This is really a first-cut response, with 2 visible possibilities...

1.  Use 2 license servers, one with 'i' licenses available for short 
jobs, and one with 'j' licenses available for longer jobs.  For i < j, 
starvation of the short jobs shouldn't occur too often, save when 
there's too many masters' students trying to get their projects done in 
time to graduate and the deadline's tomorrow.

2.  Priority queuing where short jobs have the nod, and longer jobs are 
put aside and required to temporarily relinquish licenses.  Liketo to 
require programming resources to accomplish this one.

Good question.

Gerry

Robert G. Brown wrote:
> Dear Perfect Masters of Grid Computing:
> 
> Economics is preparing to set up a small pilot cluster at Duke and the
> following question has come up.
> 
> Primary tasks:  matlab and stata jobs, run either interactively/remote
> or (more likely) in batch mode.  Jobs include both "short" jobs that
> might take 10-30 minutes run by e.g. 1-2nd year graduate students as
> part of their coursework and "long" jobs that might take hours to days
> run by more advanced students, postdocs, faculty.
> 
> Constraint:  matlab requires a license managed by a license manager.
> There are a finite number of licenses (currently less than the number of
> CPUs) spread out across the pool of CPUs.
> 
> Concern:  That long running jobs will get into the queue (probably SGE
> managed queue) and starve the short running jobs for either licenses or
> CPUs or both.  Students won't be able to finish their homework in a
> timely way because long running jobs de facto hog the resource once they
> are given a license/CPU.
> 
> I am NOT an SGE expert, although I've played with it a bit and read a
> fair bit of the documention.  SGE appears to run in FIFO mode, which of
> course would lead to precisely the sort of resource starvation feared or
> equal share mode.  Equal share mode appears to solve a different
> resource starvation problem -- that produced by a single user or group
> saturating the queue with lots of jobs, little or big, so that others
> submitting after they've loaded the queue have to wait days or weeks to
> get on.  However, it doesn't seem to have anything to do with job
> 
>>>control<< according to a policy -- stopping a long running job so that
> 
> a short running job can pass through.
> 
> It seems like this would be a common problem in shared environments with
> a highly mixed workload and lots of users (and indeed is the problem
> addressed by e.g. the kernel scheduler in almost precisely the same
> context on SMP or UP machines).  Recognizing that the license management
> problem will almost certainly be beyond the scope of any solution
> without some hacking and human-level policy, are there any well known
> solutions to this well known problem?  Can SGE actually automagically
> control jobs (stopping and starting jobs as a sort of coarse-grained
> scheduler to permit high priority jobs to pass through long running low
> priority jobs)?  Is there a way to solve this with job classes or
> wrapper scripts that is in common use?
> 
> At your feet, your humble student waits, oh masters of SGE and Grids...
> 
>     rgb
> 

-- 
Gerry Creager -- gerry.creager at tamu.edu
Network Engineering -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
Page: 979.228.0173
Office: 903A Eller Bldg, TAMU, College Station, TX 77843