[Beowulf] scheduler and perl

[Robert G. Brown]

On Tue, 1 Aug 2006, Xu, Jerry wrote:

>
> Hi, I am maintaining a cluster while lots user uses perl to submit tons of jobs
> which seems to me like abusing the system.
>
> Does everybody meet the same situation? Many user us system call in the perl to
> do "qsub", shall I ban this? I don't know exactly why it is bad, but it looks to
> me really bad. Anybody can give me a good reason to ban it with better
> explanation?

What about using perl scripts to submit jobs to a queuing system
constitutes abuse?  The whole point of a queuing system is that you can
control/monitor how much of a resource any group receives.  That is,
using it you can make it so that the resource isn't allocated first come
first serve, so somebody with an automated job submission tool "wins",
but in a prioritized queue where you can balance access no matter how
jobs are submitted.

I'd say that writing a script to manage personal job submission is
intelligence, not abuse -- it saves human time, and human time is the
most expensive resource in nearly any computing situation.  Also, for
many tasks, one sweeps some parameter(s) across a range and submit a
systematically related series of jobs.  This is clearly something that a
computer script can do FAR better than a human pecking at a keyboard.  I
actually use(d) an adaptive script that monitored the returns from
simulational jobs generating data across such a range, recomputed
mean/sigma for the data points of interest, and resubmitted jobs on a
squeaky wheel basis to uniformly drop sigma.  In that way I ended up
with a very uniform set of error bars on the data points I ultimately
fit, which in turn makes it a lot easier to get a believable fit or to
resolve problems with a fit form that might be missed by chance
otherwise.

So don't stop users from submitting tons of jobs using scripts, just
institute a policy such that those jobs have a "lowest priority" and
will be run in a fair way as the resource is available.

    rgb

>
> Sincerely
>
> Protein
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu