[Beowulf] transcode Similar Video Processing on Beowulf?
james at talkunafraid.co.uk
Tue Apr 15 04:38:42 PDT 2014
-----BEGIN PGP SIGNED MESSAGE-----
On 14/04/2014 14:13, Gavin W. Burris wrote:
> Hi, Lawrence.
> Maybe rolling something with a ZeroMQ / 0MQ would be a good idea.
> This could be a very fast way to pass many small messages that hand
> off those connections and tasks.
> I am very intrigued by this kind of queuing and think it is the
> path forward for many workflows, traditional cluster and cloud. It
> allows for asynchronous communication and fault tolerance in large
> distributed systems. I've used this to wrap a grid engine queue,
> passing python objects that execute in parallel on many nodes.
> It's cool stuff.
FWIW my $dayjob currently involves building a media analysis system
designed to run on cloud (public, private) infrastructure. ZeroMQ and
friends are not all of the answer by a long shot - there's a lot more
complexity involved in making a workflow and processing system robust
to all the failure conditions you can experience when dealing with
hundreds or thousands of nodes than simple queueing. It is a nice tool
to implement such systems with but don't underestimate the complexity
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
-----END PGP SIGNATURE-----
More information about the Beowulf