[Beowulf] SGE + policy

[Chris Dagdigian]

Here are some personal notes I've collected on license management and 
policies under Grid Engine. Perhaps they may be helpful?

FLEXlm license integration with grid engine 5.3:
http://bioteam.net/dag/sge-flexlm-integration/

Grid Engine can do very sophisticated resource allocation above and 
beyond FIFO ordering.

Your need to *stop* a long running job to let a shorter one in is a bit 
harder as graceful job premption/supension is something that normally 
takes a bit of configuring (both of the app and of Grid Engine).

There are other mechanisms in grid engine that people have used to 
prevent the 'job starvation' issue. The SGE mailing list 
users at gridengine.sunsource.net (I think you must be a member to post) or 
the searchable mailing list archives at http://gridengine.sunsource.net 
may be very helpful.

In particular you may find some of the "enterprise edition" policies 
such as 'share tree' or 'functional policy' may help. They won't boot 
running jobs unless you configure it to but the policies themselves can 
help you gain 'fairshare-by-user' or 'fairshare-by-project' style 
resource allocation which may meet your needs.

Also--

There is a new feature in Grid Engine 6.0 built into the new "Urgency 
scheduling" policy --  a configurable parameter called $WaitTimeUrgency 
that causes a jobs' urgency priority to grow the longer it has been 
sitting in pending state. I've been told that one of the drivers for 
"waitTimeUrgency" was to address pending-job-starvation issues.

The problem with Urgency Scheduling is that I don't think it has been in 
use by many people for all that long. Once 6.0 comes out of beta in a 
few weeks there may be better user reports on how Urgency scheduling is 
doing in the real world.

Since documentation is seriously lacking for Grid Engine 6.0 and the Sun 
branded version called 'N1GE 6' I've tried to write down what I've 
learned about them here:

http://bioteam.net/dag/gridengine-6-features.html


Regards,
Chris





Robert G. Brown wrote:

> Dear Perfect Masters of Grid Computing:
> 
> Economics is preparing to set up a small pilot cluster at Duke and the
> following question has come up.
> 
> Primary tasks:  matlab and stata jobs, run either interactively/remote
> or (more likely) in batch mode.  Jobs include both "short" jobs that
> might take 10-30 minutes run by e.g. 1-2nd year graduate students as
> part of their coursework and "long" jobs that might take hours to days
> run by more advanced students, postdocs, faculty.
> 
> Constraint:  matlab requires a license managed by a license manager.
> There are a finite number of licenses (currently less than the number of
> CPUs) spread out across the pool of CPUs.
> 
> Concern:  That long running jobs will get into the queue (probably SGE
> managed queue) and starve the short running jobs for either licenses or
> CPUs or both.  Students won't be able to finish their homework in a
> timely way because long running jobs de facto hog the resource once they
> are given a license/CPU.
> 
> I am NOT an SGE expert, although I've played with it a bit and read a
> fair bit of the documention.  SGE appears to run in FIFO mode, which of
> course would lead to precisely the sort of resource starvation feared or
> equal share mode.  Equal share mode appears to solve a different
> resource starvation problem -- that produced by a single user or group
> saturating the queue with lots of jobs, little or big, so that others
> submitting after they've loaded the queue have to wait days or weeks to
> get on.  However, it doesn't seem to have anything to do with job
> 
>>>control<< according to a policy -- stopping a long running job so that
> 
> a short running job can pass through.
> 
> It seems like this would be a common problem in shared environments with
> a highly mixed workload and lots of users (and indeed is the problem
> addressed by e.g. the kernel scheduler in almost precisely the same
> context on SMP or UP machines).  Recognizing that the license management
> problem will almost certainly be beyond the scope of any solution
> without some hacking and human-level policy, are there any well known
> solutions to this well known problem?  Can SGE actually automagically
> control jobs (stopping and starting jobs as a sort of coarse-grained
> scheduler to permit high priority jobs to pass through long running low
> priority jobs)?  Is there a way to solve this with job classes or
> wrapper scripts that is in common use?
> 
> At your feet, your humble student waits, oh masters of SGE and Grids...
> 
>     rgb
> 

-- 
Chris Dagdigian, <dag at sonsorol.org>
BioTeam  - Independent life science IT & informatics consulting
Office: 617-665-6088, Mobile: 617-877-5498, Fax: 425-699-0193
PGP KeyID: 83D4310E iChat/AIM: bioteamdag  Web: http://bioteam.net