[Beowulf] transcode Similar Video Processing on Beowulf?

[James Harrison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14/04/2014 14:13, Gavin W. Burris wrote:
> Hi, Lawrence.
> 
> Maybe rolling something with a ZeroMQ / 0MQ  would be a good idea.
> This could be a very fast way to pass many small messages that hand
> off those connections and tasks.
> http://zguide.zeromq.org/page:all
> 
> I am very intrigued by this kind of queuing and think it is the
> path forward for many workflows, traditional cluster and cloud.  It
> allows for asynchronous communication and fault tolerance in large
> distributed systems.  I've used this to wrap a grid engine queue,
> passing python objects that execute in parallel on many nodes.
> It's cool stuff.

FWIW my $dayjob currently involves building a media analysis system
designed to run on cloud (public, private) infrastructure. ZeroMQ and
friends are not all of the answer by a long shot - there's a lot more
complexity involved in making a workflow and processing system robust
to all the failure conditions you can experience when dealing with
hundreds or thousands of nodes than simple queueing. It is a nice tool
to implement such systems with but don't underestimate the complexity
involved!

- -- 
Cheers,
James Harrison
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQIcBAEBAgAGBQJTTRpCAAoJENTyYHL8dmp9P9IQAJc6zSv0Soo43oOnnlAEm1Oe
+DmpLxsh5a/TkM6JqoEAhzf9MtqAxeznxpSJ0CcJ13K5HidyTqqGmB4nEQV0f3Sd
BkTR8E+0KSHIPeHQBikcxMJd5eRm++JvZ9tNaIbqwpDbnJ688S6hoin7dcpAu5J5
Qq+psvazgrAaOaffFid4uBpI18GJh3+kbdEQ6Muda8LQre2EzISA+Q7zTRMnpuN3
bm9cOMoq5ricpbC9MImhJII9M1LpnSUlaqP/yCYqM/Fi29CKNikDev1hWTHAzGn4
8qc7JEhHVpyf2cCAxjOsI7XTDnk+fLoBu9reW+3zn7TjeiTUiAmRTDZ8hMKV/D/7
GGfVvgBTDuIK9RgfpHz0LfaRMi4rhGcUfxqfbSZRQb/XJQuqRJRh98oWY1oY4+do
SaoAZwaS/KnhVvj5iYh9c/JEkc23sBhfHcQccpcbaXeFfKn1hVw7nvjvK+Fw2Z8r
Ed/xZK9GSS0rw7rMGkL3o6AvNvpgrwojPfQ9yxbd5K1M1+kpZOllh1XNYxY/2Nyz
95Wu715w7BuRDXeykduz7CRRzzdHeJUhvPxzAXL1cXxETFOezS/oD8SaO5HH8uAZ
ZKi2/BaoX18ctwxNBp4FE4SbVbs9112VdA6CmYtroRCR3ZCJK1DaJG6LyWmaVqSQ
1zzcVX91ozPOYNd2ld38
=n39x
-----END PGP SIGNATURE-----