Myrinet setup (was: RE: [Beowulf])

[William Burke]

Hi Reuti,

Quoting Reuti [reuti at staff.uni-marburg.de]:

<snip>
> First one additional question (before I route you in the wrong direction):

> is it necessary for you to have a shared $TMPDIR for SGE? ...More common 
> and faster is to use the local /tmp on the nodes for this

Actually there is no reason that I should have a shared $TMPDIR for SGE,
except that I read some where (I am not sure where)that it was recommended
to share the $TMPDIR. However I just seached the SGE and N1GE docs I found
no evidence supporting that notion and now thinking about it setting $TMPDIR
to local /tmp would simplfy things. Thxs

Regards,
William

-----Original Message-----
From: Reuti [mailto:reuti at staff.uni-marburg.de] 
Sent: Sunday, March 27, 2005 7:09 AM
To: William Burke
Cc: users at gridengine.sunsource.net; beowulf at beowulf.org; dag at sonsorol.org;
'John Hearns'
Subject: Myrinet setup (was: RE: [Beowulf])

Hi Will,

Quoting William Burke <wburke999 at msn.com>:

<snip>
> I have but I do not see my name yet? How long is the verification process?

did you register as an observer - AFAIK you can post to the SGE list without

being registered.

<snip>
> The mpi directory's mpich.template doesn't use mpirun.ch_gm so how does it
> know what version of mpirun to use? If I use the mpi what changes do I
have
> to make?

You can have more than one MPI implementation installed in your cluster, and
it 
may need some planning to set up the correct $PATH for each of the 
implementations you want to use (and the this way located mpirun must fit to

your used version of MPI during compilation of your program). You may use a 
"which mpirun" to check it in your job script.

Also the supplied sge_mpirun will not use any Myrinet version on it's own - 
it's just a wrapper to the mpirun you set in the PE, so that you don't have
to 
specify the usual options 'mpirun -machinefile $TMPDIR/machines -np $NSLOTS 
mypgm'.

I must admit: seems that the Myrinet stuff was more for 5.3 and not updated,
as 
in 6.0 you can have more than one line for "start_proc_args" in your PE 
definition - so it just grabs the last \ in the first and only line
beginning 
with "start_proc_args" line as mpirun command - will give the error message
you 
got, that "\" is not existing.

As I said: we can use the default MPICH integration also for Myrinet and 
proceed this way.

> >> Can you please give more details of your queue and PE setup (qconf
> -sq/sp

Thx, I will keep the stuff. First one additional question (before I route
you 
in the wrong direction): is it necessary for you to have a shared $TMPDIR
for 
SGE? This is the one you set in your queue configuration (tmpdir 
/WEMS/grid/tmp) and seems for now to be on a file server.

More common and faster is to use the local /tmp on the nodes for this (you
are 
right: SGE want to create there a directory for this job and some file for
its 
own usage - but you are free to use this directory $TMPDIR also in your job 
script). It will be created for your job, and cleanly deleted after the job,
so 
you won't have any leftover files.

Cheers - Reuti