[Beowulf] transcode Similar Video Processing on Beowulf?

[Gavin W. Burris]

Hi, Lawrence.

Maybe rolling something with a ZeroMQ / 0MQ  would be a good idea.  This
could be a very fast way to pass many small messages that hand off those
connections and tasks.  http://zguide.zeromq.org/page:all

I am very intrigued by this kind of queuing and think it is the path
forward for many workflows, traditional cluster and cloud.  It allows
for asynchronous communication and fault tolerance in large distributed
systems.  I've used this to wrap a grid engine queue, passing python
objects that execute in parallel on many nodes.	 It's cool stuff.	

Cheers.

On Mon 04/14/14 08:59AM -0400, Lawrence Stewart wrote:
> This inquiry brings up a subject that has long puzzled me.  In my (limited) experience,
> resource managers like SLURM (which I've used) are pretty good at handling flows
> of independent MPI jobs that set up, run, and exit.
> 
> I don't know what to use to manage a pool of servers such as a collection of, say, video
> transcoders.  A particular instance of the transcoder might be able to handle up to
> N streams, but because the streams start and stop at different times, the server is always
> doing something.
> 
> Are there standard packages to manage keeping a collection of daemon servers running on
> a cluster, and then managing the connections handled by those servers?
> 
> -Larry
> 
> On 2014, Apr 14, at 2:32 AM, Yew, Tze Hui <tze.hui.yew at intel.com> wrote:
> 
> > Hi Michael Will,
> >  
> > I’m TH and am interested with this http://www.beowulf.org/pipermail/beowulf/2005-January/011626.html. I’m currently looking at a solution to launch an object detection app on Host, with the GUI running on Host and the compute nodes doing all the video processing and analytics part. I see this solution is similar to my approach but I couldn’t find any information on how to setup. May I know where to more info about this? As I googled, Cinelerra is a video processing editor, which has nothing to do with clustering. Can you guide me on this? Appreciate your help.
> >  
> > Thanks.
> >  
> > FYI, I have tried to send jobs(with graphics) using MPI previously but failed, seems like MPI only suitable for data processing but not video processing.
> >  
> >  
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visithttp://www.beowulf.org/mailman/listinfo/beowulf
> 

> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


-- 
Gavin W. Burris
Senior IT Project Leader
Research and Innovation Team
The Wharton School
University of Pennsylvania