Myrinet setup (was: RE: [Beowulf])
wburke999 at msn.com
Sun Mar 27 07:00:56 PST 2005
Quoting Reuti [reuti at staff.uni-marburg.de]:
> First one additional question (before I route you in the wrong direction):
> is it necessary for you to have a shared $TMPDIR for SGE? ...More common
> and faster is to use the local /tmp on the nodes for this
Actually there is no reason that I should have a shared $TMPDIR for SGE,
except that I read some where (I am not sure where)that it was recommended
to share the $TMPDIR. However I just seached the SGE and N1GE docs I found
no evidence supporting that notion and now thinking about it setting $TMPDIR
to local /tmp would simplfy things. Thxs
From: Reuti [mailto:reuti at staff.uni-marburg.de]
Sent: Sunday, March 27, 2005 7:09 AM
To: William Burke
Cc: users at gridengine.sunsource.net; beowulf at beowulf.org; dag at sonsorol.org;
Subject: Myrinet setup (was: RE: [Beowulf])
Quoting William Burke <wburke999 at msn.com>:
> I have but I do not see my name yet? How long is the verification process?
did you register as an observer - AFAIK you can post to the SGE list without
> The mpi directory's mpich.template doesn't use mpirun.ch_gm so how does it
> know what version of mpirun to use? If I use the mpi what changes do I
> to make?
You can have more than one MPI implementation installed in your cluster, and
may need some planning to set up the correct $PATH for each of the
implementations you want to use (and the this way located mpirun must fit to
your used version of MPI during compilation of your program). You may use a
"which mpirun" to check it in your job script.
Also the supplied sge_mpirun will not use any Myrinet version on it's own -
it's just a wrapper to the mpirun you set in the PE, so that you don't have
specify the usual options 'mpirun -machinefile $TMPDIR/machines -np $NSLOTS
I must admit: seems that the Myrinet stuff was more for 5.3 and not updated,
in 6.0 you can have more than one line for "start_proc_args" in your PE
definition - so it just grabs the last \ in the first and only line
with "start_proc_args" line as mpirun command - will give the error message
got, that "\" is not existing.
As I said: we can use the default MPICH integration also for Myrinet and
proceed this way.
> >> Can you please give more details of your queue and PE setup (qconf
Thx, I will keep the stuff. First one additional question (before I route
in the wrong direction): is it necessary for you to have a shared $TMPDIR
SGE? This is the one you set in your queue configuration (tmpdir
/WEMS/grid/tmp) and seems for now to be on a file server.
More common and faster is to use the local /tmp on the nodes for this (you
right: SGE want to create there a directory for this job and some file for
own usage - but you are free to use this directory $TMPDIR also in your job
script). It will be created for your job, and cleanly deleted after the job,
you won't have any leftover files.
Cheers - Reuti
More information about the Beowulf