[Beowulf] transcode Similar Video Processing on Beowulf?

[rf at q-leap.de]

>>>>> "James" == James Harrison <james at talkunafraid.co.uk> writes:

    James> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

    James> On 14/04/2014 14:13, Gavin W. Burris wrote:
    >> Hi, Lawrence.
    >>
    >> Maybe rolling something with a ZeroMQ / 0MQ would be a good idea.
    >> This could be a very fast way to pass many small messages that
    >> hand off those connections and tasks.
    >> http://zguide.zeromq.org/page:all
    >>
    >> I am very intrigued by this kind of queuing and think it is the
    >> path forward for many workflows, traditional cluster and cloud.
    >> It allows for asynchronous communication and fault tolerance in
    >> large distributed systems.  I've used this to wrap a grid engine
    >> queue, passing python objects that execute in parallel on many
    >> nodes.  It's cool stuff.

    James> FWIW my $dayjob currently involves building a media analysis
    James> system designed to run on cloud (public, private)
    James> infrastructure. ZeroMQ and friends are not all of the answer
    James> by a long shot - there's a lot more complexity involved in
    James> making a workflow and processing system robust to all the
    James> failure conditions you can experience when dealing with
    James> hundreds or thousands of nodes than simple queueing. It is a
    James> nice tool to implement such systems with but don't
    James> underestimate the complexity involved!

FWIW another note about ZeroMQ:

With version 4 it now also has encryption (fast, secure elliptic-curve
crypto based on NaCl lib) and authentification. Our cluster management
framework QluMan (part of Qlustar / qlustar.com) uses ZMQ since almost 2
years, and with curve based encryption (which we're already using
successfully) it is now ready to securely manage stuff in the cloud. Our
secure remote execution engine is also based on ZMQ/Curve.

Can only recommend it.

Roland