[Beowulf] Prioritising Pending Jobs from different queues

[Chris Dagdigian]

Hi Jenno,

The users at gridengine.sunsource.net mailing list is where quite a few  
SGE guru's hang out and provide helpful suggestions.

Grid Engine is just different enough from PBS/Torque/LSF that mapping  
things into clear queues sorted by priority is <ahem> an interesting  
exercise. Generally speaking, the preferred behavior with SGE is that  
users should request the resources that their job needs in order to  
be successful and the SGE scheduler will actually sort out the most  
preferred queue. The term 'queue' means different things on an LSF  
system vs a SGE system, and SGE best practices gently discourage most  
users from thinking of queues at all.

One possible suggestion that would get you the behavior you require  
would be to use the SGE 'urgency' policy mechanism. The urgency  
policy was originally put in place to allow system managers to  
maximize the utilization of "expensive" assets like commercial  
software licenses or monster leased supercomputers etc. Jobs  
requesting the "expensive" assets would receive a boost in scheduler  
preference. The amount of the boost depends on the number of urgency  
tickets and the overall weight that the Urgency policy has within the  
scheduling algorithm.

I suspect that if you enabled the Urgency policy and assigned some of  
those policy entitlements to your "high priority" queue then any jobs  
you have that specifically ask for the high priority queue would be  
jumped to the top of the pending list for dispatch into the first  
available high priority job slot.

There are many ways to do these sorts of things within SGE so no  
promises that this is the "best" method. The SGE gurus would probably  
ask why you even need to have 2 queues at all, especially as you seem  
mostly concerned with designating some jobs as "more important" than  
others which is something that can generally be done within a single  
queue pretty easily using some of the other SGE policy based resource  
allocation mechanisms.

Regards,
Chris



On Dec 5, 2005, at 7:34 AM, Jenno Claesson wrote:

> Hi,
> Just re-asking a question for which I forgot to mention I was using  
> the Sun Grid Engine.
>
> My question surrounds the submission of jobs into two separate queues,
> assuming I have only one host on which to run the jobs.
>
> If I have two queues, is there a way that I can have jobs submitted  
> in one
> queue, run before jobs that were previously submitted in another  
> queue.  I
> am happy to wait for any currently running jobs to finish, and as  
> such I
> don't really want to use subordinate_list, because I do not want to  
> effect
> any running jobs.  All I want to do, is ensure that the job in my  
> "high
> priority" queue, is the next job to be run from the "Pending Jobs"  
> list.
>
> I also don't want to have to use the -p option at submit time.   
> Essentially,
> all I want is a high-priority queue, and a low priority queue.
>