<!DOCTYPE HTML PUBLIC "-//W3O//DTD W3 HTML 2.0//EN">
<html>
<head>
<title>SciClone Man Pages: pbslam</title>
</head>
<BODY BGCOLOR="white" TEXT="black" LINK="#000099" VLINK="#660099">
<p align=right><i><font size=-1><a href="../index.html">SciClone Local Manual</a></font></i>
<p>
<h2>pbslam</h2>
<hr>
<dl>
<dt>
<h3>Purpose</h3>
<dd>Run a LAM MPI program under PBS. </dd>
<dt>
<h3>Synopsis</h3>
<dd> <b>exec pbslam</b> [<b>-dfghnOtTvx</b>] [<b>-c</b> np] [<b>-C</b> load
| <b>-X</b> load] [<b>-D</b> | <b>-W</b> dir] program [args...]
</dd>
<dt>
<h3>Description</h3>
<dd>The PBS job scheduling system allocates resources for parallel programs,
but does not provide the system-specific procedures for actually initiating
and executing parallel programs on those resources. <i>pbslam</i> provides
such an interface between PBS and the LAM MPI runtime system, including the
following services:<br>
<ul>
<li>Mapping of processes onto assigned processors (by constructing a LAM
application schema).
<li>Initialization of the LAM runtime environment (<i>lamboot</i>).
<li>Execution of the LAM program (via <i>mpirun</i>).
<li>Shutting down the LAM runtime system (equivalent to <i>wipe</i>, only
faster).
<li>Intercepting abnormal termination conditions (<i>qdel</i> requests,
over-limit run times, keyboard interrupts, etc.) in order to clean up
LAM processes before aborting a PBS job.
</ul>
<p>In order to properly intercept termination signals, <i>pbslam</i> must
be <i>exec</i>'ed, replacing the shell which invokes it. <i>pbslam</i> checks
for this, and will complain if it is not in the proper location within the
PBS process hierarchy.
<p>For added flexibility, <i>pbslam</i> provides two different strategies
for mapping processes onto processors; these are described in detail in
the section on <a href="#mapping">Process Mapping</a>. Which strategy is
best depends on the requirements of the application, the number and type
of nodes requested for the job, and the number of processes which will be
run on those nodes.
</dd>
<dt>
<h3>Arguments</h3>
<dd>
<dl>
<dt><b>-c</b> <i>np</i>
<dd>Run <i>np</i> copies of <i>program</i> on the assigned nodes. If this
option is not specified, one process is assigned to each PBS virtual processor.
<br>
<dt><b>-C</b> <i>load</i>
<dd>Before starting the program, check the CPU utilization on each node
assigned to the job, and report any which exceed <i>load</i>. <i>load</i>
should be a decimal fraction in the range from 0.0 to 1.0. By default,
no checking is done. <b>-C</b> and <b>-X</b> are mutually exclusive. A
certain amount of system-related background activity is unavoidable, so
the minimum useful value for <i>load</i> is probably in the 0.01-0.02
range.<br>
<dt><b> -d</b> </dt>
<dd>Use indirect communication via LAM daemons. By default, processes communicate
directly with each other (c2c mode), bypassing the LAM daemons.<br>
<dt><b>-D</b>
<dd>Use the directory which contains <i>program</i> as the working directory
for LAM processes. By default, <i>pbslam</i> runs <i>program</i> in the
directory from which it is invoked (i.e., the current working directory).
The same directory pathname is used on all nodes. <b>-D</b> and <b>-W</b>
are mutually exclusive.<br>
<dt><b>-f</b>
<dd>Do not configure LAM's standard I/O descriptors. Output from remote
processes is directed to /dev/null. By default, stdout and stderr from
remote processes are routed back to stdout and stderr of LAM's <i>mpirun</i>
command, which in turn is routed to stdout and stderr of <i>pbslam</i>.
<br>
<dt><b>-g</b>
<dd>Enable LAM's Guaranteed Envelope Resources (GER) mode. GER is disabled
by default.<br>
<dt><b>-h</b>
<dd>Print a help message, listing the available options with a brief description
of each.<br>
<dt><b>-n</b>
<dd>Use a node-order strategy (explained <a href="#mapping">below</a>) for
mapping processes to processors.<br>
<dt><b>-O</b>
<dd>Enable data conversion for heterogeneous environments with mixed byte
orders. By default, data conversion is disabled.<br>
<dt><b>-t</b>
<dd>Enable LAM's trace generation capability, with tracing initially turned
off. By default, tracing is disabled. Mutually exclusive with <b>-T</b>.<br>
<dt><b>-T</b>
<dd>Enable LAM's trace generation capability, with tracing initially turned
on. Mutually exclusive with <b>-t</b>.<br>
<dt><b>-v</b>
<dd>Verbose mode. Enable verbose option on LAM commands, and generate additional
output about the progress of <i>pbslam</i>, as well as a listing of allocated
nodes and the mapping of processes to processors.<br>
<dt><b>-W</b> <i>dir</i>
<dd>Use <i>dir</i> as the working directory for LAM processes. <b>-W</b>
and <b>-D</b> are mutually exclusive.<br>
<dt><i>program</i>
<dd>Name of the LAM MPI program to be invoked via <i>mpirun</i>. If a full
pathname is not given, the current search path ($PATH environment variable)
is used to locate <i>program</i>.<br>
<dt><i>args...</i>
<dd>Arguments for <i>program</i>.
</dl>
</dd>
<dt>
<h3><a name="mapping"></a>Process Mapping</h3>
<dd>When PBS is configured, each node in the system is assigned one or more
<i>virtual processors </i>(or <i>VP</i>'s, for short).<i> </i>On SciClone,
the number of PBS virtual processors on each node is identical to the number
of physical processors on that node (except for the front end, which only
allows PBS to use one of its two processors). PBS then allocates virtual processors
to jobs, based on the resource requirements specified by the <i>qsub</i> command.
The hostnames of each virtual processor allocated to a job are available at
runtime in a file specified by the PBS_NODEFILE environment variable. The
order in which hosts are listed in this file correpsonds to the order in which
they are requested by the "-l nodes=" option of <i>qsub</i>.
<p> <i>pbslam</i> supports two different schemes for mapping LAM processes
onto PBS virtual processors. We call one of these schemes the "VP order",
and the other "node order". By default, <i>pbslam</i> uses VP
order; node order is invoked with the <b>-n</b> option. The contents of
PBS_NODEFILE, as well as the mapping of processes onto nodes, is displayed
on stdout when the <b>-v</b> option is specified.
<p> <i>VP Order:</i> Processes are assigned one per VP in the order listed
in PBS_NODEFILE. (Note that there may be more than one VP per node.) If
the number of processes requested by the <b>-c</b> option is larger than
the number of VPs allocated to the job, then wrap around to the beginning
of the VP list, assigning an additional process to each VP. This procedure
repeats until all processes have been assigned.
<p> <i>Node Order:</i> Processes are assigned one per node, wrapping around
until all of the VP slots on all of the nodes are filled. If the number
of processes requested by the <b>-c</b> option is larger than the number
of VPs allocated to the job, wrap around and assign an additional process
to each node (rather than VP), repeating until all processes have been assigned.
<p> <i>Example 1:</i> Our first example illustrates the difference in assignment
strategies for a job which maps 12 processes onto 8 virtual processors which
are spread across four nodes. Assume the following PBS job request:
<blockquote><code>qsub -l nodes=2:single+1:dual:ppn=2+1:quad:ppn=4<br>
pbslam -c 12 myprog</code></blockquote>
<p> On SciClone, the resulting PBS_NODEFILE might look like:
<blockquote><code>ty01<br>
ty02<br>
tn01<br>
tn01<br>
hu01<br>
hu01<br>
hu01<br>
hu01</code></blockquote>
<p> With the default VP order, processes would be mapped as follows:
<blockquote><code>p0 -> ty01<br>
p1 -> ty02<br>
p2 -> tn01<br>
p3 -> tn01<br>
p4 -> hu01<br>
p5 -> hu01<br>
p6 -> hu01<br>
p7 -> hu01<br>
p8 -> ty01<br>
p9 -> ty02<br>
p10 -> tn01<br>
p11 -> tn01 </code></blockquote>
<p> If the <b>-n</b> option had been used on the <i>pbslam</i> command, the
mapping would instead be:
<blockquote><code>p0 -> ty01<br>
p1 -> ty02<br>
p2 -> tn01<br>
p3 -> hu01<br>
p4 -> tn01<br>
p5 -> hu01<br>
p6 -> hu01<br>
p7 -> hu01<br>
p8 -> ty01<br>
p9 -> ty02<br>
p10 -> tn01<br>
p11 -> hu01 </code></blockquote>
<p> <i>Example 2:</i> The <b>-n</b> option is particularly useful if you want
exclusive use of a multi-processor node, but only want to use a subset of
the processors on each node. The following example allocates all 16 processors
on 8 dual-cpu nodes (thereby ensuring that the job has exclusive use of
the nodes), but only assigns one process to each node:
<blockquote><code>qsub -l nodes=8:compute:dual:ppn=2<br>
pbslam -c 8 myprog</code></blockquote>
</dd>
<dt>
<h3>Exit Status</h3>
<dd>
If the LAM job runs to completion, <i>pbslam</i> returns the exit status from
the <i>mpirun</i> command. If <i>pbslam</i> terminates by catching a signal,
it returns the signal number. If <i>pbslam</i> detects any other error condition,
it returns a non-zero value.
</dd>
<dt>
<h3>Bugs and Limitations</h3>
<dd> <i>pbslam</i> does not provide access to all of the options and features
supported in LAM. In particular, there is no way to directly specify different
executables for different processes, although various workarounds can be imagined.
<p> Circumstances may arise in which <i>pbslam</i> (or any other PBS job,
for that matter) might not be able to find and kill all of the processes
belonging to it. If a pristine execution environment is essential, additional
checks (beyond <b>-C</b> or <b>-X</b>) may be needed to ensure that no stray
processes reside on a node.
</dd>
<dt>
<h3>Related Topics</h3>
<ul>
<li><a href=pbsmpich.html target="_self">pbsmpich</a>, <a href=pbspvm.html target="_self">pbspvm</a>
<li><a href=../../../SciCloneUserGuide.html target="_top">SciClone User's
Guide</a>
</ul>
</dl>
<hr>
<font size=-1> </font>
</body>
</html>