[Beowulf] IB troubles - mca_mpool_openib_register

[Greg Lindahl]

> The problem is that when I run with nodes=64:ppn=2 and only use -np 64 
> with my openmpi (v 1.0.2 gcc compiled), it still runs fine, but when I 
> run with -np 65 I get megabytes of error messages and the job never 
> completes.  The errors all look like this:
> 
> mca_mpool_openib_register: ibv_reg_mr(0x2a96641000,1060864)
> failed with error: Cannot allocate memory

.. which means you need to raise that memlock limit higher than your
current 8192, which is only 8 megabytes. Most MPIs on IB lock lots of
memory, more if you use more processes, and more for apps that send
large messages.

-- greg