[Beowulf] SGE web frontends

[Robert G. Brown]

On Fri, 4 Feb 2005, Joe Landman wrote:

> 
> 
> Robert G. Brown wrote:
> > On Wed, 2 Feb 2005, Matthew Pratola wrote:
> > 
> > 
> >>Hi all,
> >>
> >>Can anyone recommend a simple web frontend for submitting SGE jobs?
> > 
> > 
> >   http://www.globus.org/
> > 
> > One stop shopping.
> 
> Did I miss something?  Was a tongue planted in cheek with this reply?

Actually, it was a reply I snapped off on my way out the door on the
edge of late for teaching.

Let me reconsider my answer.

You don't like yes/globus, how about "no".

At least if you mean really really simple by simple.

I would argue that a cluster designed to run primarily embarrassingly
parallel jobs, fronted by a web portal/interface, is a not uncommon form
of a grid, although perhaps the definition is large enough to include a
union of such clusters or some more general structure (certainly access
to other kinds of resources than strictly "a cluster"). So I read this
question as "I want to make my local cluster into a grid, so users
faraway with no direct LAN accounts or access can submit jobs into my
local SGE queue after being properly authenticated".  And, of course, be
notified (with messages) when the jobs crash or terminate normally,
facilitate data transfer and resource allocation requests, etc.  Not
exactly simple...

Globus TK is as I understand it a toolkit from which one can build a web
interface for generalized remote task submission to "a grid".  It has to
have lots of moving parts to do that well -- just AUTHENTICATING data
transfer and job execution via a web interface isn't really terribly
"simple", becaues to do it decently generally requires e.g. stuff like
kerberos, ssl, ssh that aren't terribly simple either.  So I definitely
failed on the "simple" bit.

However, simple or not, I believe that Globus does contain the
components to do what you want -- provide a very generic web interface
for people far away who don't share any LAN components such as mounted
filespace, authentication/userid mappings, etc to transfer data and job
execution instructions to a system.  That system, if it is a front end
running SGE and/or stuff like condor (policy, load balance, batch job
tools) can then put the job into a queue, run it, and let globus know
when it is finished so it can tell the original user.

If you look over just their security layer (GSI -- Grid Security
Infrastructure) you rapidly come to realize that to run any sort of
remote job execution service you NEED most of its components --
authentication (including a Certificate Authority CA), encryption
(public/private key, managed with certificates), permissions, etc.  Some
grid designs I've seen use just this component of Globus and use other
tools (like PBS or SGE or custom designed stuff) for other components.

Ian Foster seems to have a list of at least some of the major grid
projects around the world -- enough to be able to google on them by
name -- here:

  http://www-fp.mcs.anl.gov/~foster/grid-projects/

Perhaps you can find a reusable interface at one of their project
websties.  You can also check out e.g. the Grid Portal Development Kit:

  http://doesciencegrid.org/projects/GPDK/

or The Grid Portal Kit:

  https://gridport.npaci.edu/

or the Open Grid Computing Environment:

  http://www.ogce.org/index.php

all of which I believe use globus as at least part of their middleware
for e.g. authentication etc.  Some of these are (e.g. the DOE's GPDK)
currently unsupported although still available and possibly still
reasonably functional.  I don't really know the status of the rest of
them, and I doubt that this is all of them.

So you're right, I should have answered "no" because it isn't simple to
offer a web interface to any active service, ESPECIALLY one that permits
a remote user to upload arbitrary programs for execution on arbitrary
data of arbitrary size where authentication, encryption, data transport,
and remote job management become absolutely essential components of the
solution.

AFAIK, Globus is one of the if not the only middleware toolkits of
choice for people who run the big grids -- they probably write their own
actual web portal, but they use Globus to do at least some of the heavy
lifting that goes on behind the scenes.  Maybe one of the "portal
projects" above (all open source) will be of use in setting up a
"simple" portal to your cluster, but be aware that the problem itself is
far from simple.

However, I could be wrong and as always cherish being corrected.

   rgb

> 
> As far as I know there are very few web interfaces to running SGE (or 
> LSF, or ...) jobs.  If I am wrong please do provide links/references.
> 
> Globus is not a web interface (last I checked), but a large group of 
> middleware to manage something that looks a lot closer to the definition 
> of a grid than SGE.  SGE is a job scheduler (with a name "engineered" to 
> make you think it is a one-stop-shop as a grid-in-a-box).
> 
> My company is interested in (and we are developing) web portals for end 
> user cluster work, so if you know of any, we would like to hear about 
> them.  Good open-source platforms that are current/supported could be 
> worth looking at (and will save us time/development effort).  There seem 
> to be lots of bits of abandonware in the grid portal/user-interface 
> area.  We don't want to re-invent wheels, but at the same time, we don't 
> want to adopt abandoned ones either.
> 
> Joe
> 
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu