[Beowulf] SGE + policy

[Robert G. Brown]

On Thu, 27 May 2004, Chris Dagdigian wrote:

Thanks, some very useful stuff here, especially the flexlm stuff.

> 
> Here are some personal notes I've collected on license management and 
> policies under Grid Engine. Perhaps they may be helpful?
> 
> FLEXlm license integration with grid engine 5.3:
> http://bioteam.net/dag/sge-flexlm-integration/
> 
> Grid Engine can do very sophisticated resource allocation above and 
> beyond FIFO ordering.
> 
> Your need to *stop* a long running job to let a shorter one in is a bit 
> harder as graceful job premption/supension is something that normally 
> takes a bit of configuring (both of the app and of Grid Engine).

Ya, it isn't just manipulation of the queue.  It is the difference
between a real scheduler that dispenses time-slices to a mix of tasks,
some sleeping, some queued to be run, so that all tasks get time slices
fairly whether they complete in a single slice or 10^8 slices, and one
that puts a 10^8 slice task on to run to completion if and when it makes
it to the top of the queue and never mind the jobs that then have to
wait weeks to run.

I'll look over the archives and see what I can see, and again, really
appreciate the links and comments!

   rgb

> There are other mechanisms in grid engine that people have used to 
> prevent the 'job starvation' issue. The SGE mailing list 
> users at gridengine.sunsource.net (I think you must be a member to post) or 
> the searchable mailing list archives at http://gridengine.sunsource.net 
> may be very helpful.
> 
> In particular you may find some of the "enterprise edition" policies 
> such as 'share tree' or 'functional policy' may help. They won't boot 
> running jobs unless you configure it to but the policies themselves can 
> help you gain 'fairshare-by-user' or 'fairshare-by-project' style 
> resource allocation which may meet your needs.
> 
> Also--
> 
> There is a new feature in Grid Engine 6.0 built into the new "Urgency 
> scheduling" policy --  a configurable parameter called $WaitTimeUrgency 
> that causes a jobs' urgency priority to grow the longer it has been 
> sitting in pending state. I've been told that one of the drivers for 
> "waitTimeUrgency" was to address pending-job-starvation issues.
> 
> The problem with Urgency Scheduling is that I don't think it has been in 
> use by many people for all that long. Once 6.0 comes out of beta in a 
> few weeks there may be better user reports on how Urgency scheduling is 
> doing in the real world.
> 
> Since documentation is seriously lacking for Grid Engine 6.0 and the Sun 
> branded version called 'N1GE 6' I've tried to write down what I've 
> learned about them here:
> 
> http://bioteam.net/dag/gridengine-6-features.html
> 
> 
> Regards,
> Chris
> 
> 
> 
> 
> 
> Robert G. Brown wrote:
> 
> > Dear Perfect Masters of Grid Computing:
> > 
> > Economics is preparing to set up a small pilot cluster at Duke and the
> > following question has come up.
> > 
> > Primary tasks:  matlab and stata jobs, run either interactively/remote
> > or (more likely) in batch mode.  Jobs include both "short" jobs that
> > might take 10-30 minutes run by e.g. 1-2nd year graduate students as
> > part of their coursework and "long" jobs that might take hours to days
> > run by more advanced students, postdocs, faculty.
> > 
> > Constraint:  matlab requires a license managed by a license manager.
> > There are a finite number of licenses (currently less than the number of
> > CPUs) spread out across the pool of CPUs.
> > 
> > Concern:  That long running jobs will get into the queue (probably SGE
> > managed queue) and starve the short running jobs for either licenses or
> > CPUs or both.  Students won't be able to finish their homework in a
> > timely way because long running jobs de facto hog the resource once they
> > are given a license/CPU.
> > 
> > I am NOT an SGE expert, although I've played with it a bit and read a
> > fair bit of the documention.  SGE appears to run in FIFO mode, which of
> > course would lead to precisely the sort of resource starvation feared or
> > equal share mode.  Equal share mode appears to solve a different
> > resource starvation problem -- that produced by a single user or group
> > saturating the queue with lots of jobs, little or big, so that others
> > submitting after they've loaded the queue have to wait days or weeks to
> > get on.  However, it doesn't seem to have anything to do with job
> > 
> >>>control<< according to a policy -- stopping a long running job so that
> > 
> > a short running job can pass through.
> > 
> > It seems like this would be a common problem in shared environments with
> > a highly mixed workload and lots of users (and indeed is the problem
> > addressed by e.g. the kernel scheduler in almost precisely the same
> > context on SMP or UP machines).  Recognizing that the license management
> > problem will almost certainly be beyond the scope of any solution
> > without some hacking and human-level policy, are there any well known
> > solutions to this well known problem?  Can SGE actually automagically
> > control jobs (stopping and starting jobs as a sort of coarse-grained
> > scheduler to permit high priority jobs to pass through long running low
> > priority jobs)?  Is there a way to solve this with job classes or
> > wrapper scripts that is in common use?
> > 
> > At your feet, your humble student waits, oh masters of SGE and Grids...
> > 
> >     rgb
> > 
> 
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu