[Beowulf] transcode Similar Video Processing on Beowulf?

[Gavin W. Burris]

On Tue 04/15/14 12:38PM +0100, James Harrison wrote:
> On 14/04/2014 14:13, Gavin W. Burris wrote:
> > Hi, Lawrence.
> > 
> > Maybe rolling something with a ZeroMQ / 0MQ  would be a good idea.
> > This could be a very fast way to pass many small messages that hand
> > off those connections and tasks.
> > http://zguide.zeromq.org/page:all
> > 
> > I am very intrigued by this kind of queuing and think it is the
> > path forward for many workflows, traditional cluster and cloud.  It
> > allows for asynchronous communication and fault tolerance in large
> > distributed systems.  I've used this to wrap a grid engine queue,
> > passing python objects that execute in parallel on many nodes.
> > It's cool stuff.
> 
> FWIW my $dayjob currently involves building a media analysis system
> designed to run on cloud (public, private) infrastructure. ZeroMQ and
> friends are not all of the answer by a long shot - there's a lot more
> complexity involved in making a workflow and processing system robust
> to all the failure conditions you can experience when dealing with
> hundreds or thousands of nodes than simple queueing. It is a nice tool
> to implement such systems with but don't underestimate the complexity
> involved!
> 

Yes!  No doubt!  The "simple" queues presuppose a massive distributed
system to take advantage of.  Bonus points if that system can
interchangeably be an in-house cluster or major cloud provider.

I would be very interested to hear what your preferred tools and APIs
are for your analysis system.  I can easily default to the job script
and qsub workflow, but restful cloud APIs and simple queues seem to be
next-level for some workflows.

Cheers,
-- 
Gavin W. Burris
Senior IT Project Leader
Research and Innovation Team
The Wharton School
University of Pennsylvania