[Beowulf] Reliable Job Queueing and Notification

Peter St. John peter.st.john at gmail.com
Wed Oct 17 11:31:23 PDT 2007


Well, you could make a mail recipient that belongs to your web service. Then
use the existing email notification to cc your service, with appropriate
keywords in subject line, then use the "you have mail" event to trigger
parsing the keyword and matching against a current user connection.

It could be pretty klugy, but OTOH there is a guy somewhere in the universe
who uses your same IIS/ASP/Whatever setup and who would do that in one line
of perl. Not me though.
Peter


On 10/16/07, Sean Ward <SeanWard at msn.com> wrote:
>
> I've started work on a web service which contains several potentially
> long running processing steps (molecular dynamics), which are perfect to
> farm out to the fairly large (90 node) Beowulf I have access to. The
> primary issue is translating requests from the event driven web service,
> to job queues, and back again upon completion. Specifically, the major
> queuing systems I have immediate access to (Sun Grid Engine and Condor)
> only support e-mail based notification of job completion. Starting jobs
> isn't an issue, as my service can simply ssh over and execute shell
> scripts as needed to start things up, the problem is reliably being
> informed when the jobs fail or complete, via any programmatic method
> (such as executing a shell script, calling a web service via SOAP/etc,
> or an asynchronous message library). My other problem, ensuring that
> these web service requests don't starve in house jobs on the Beowulf is
> easily handled via the priority levels built into all the various job
> managers, although being able to checkpoint a long running job would be
> a plus (such as is supported by Condor).
>
> I am currently investigating modifications to either Condor (more
> complex to update, but checkpoint is useful) or Ruby Queue (very easy to
> update for reliable notification) to solve this issue, but wanted to be
> sure I wasn't overlooking any existing solutions to programmatic based
> queuing and receiving notifications on jobs in a Beowulf environment...
>
> -Sean
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20071017/e45ae28b/attachment.html>


More information about the Beowulf mailing list