[Beowulf] IB troubles - mca_mpool_openib_register

[Bill Wichser]

You nailed it Greg.  Thanks!

Greg Lindahl wrote:
>>The problem is that when I run with nodes=64:ppn=2 and only use -np 64 
>>with my openmpi (v 1.0.2 gcc compiled), it still runs fine, but when I 
>>run with -np 65 I get megabytes of error messages and the job never 
>>completes.  The errors all look like this:
>>
>>mca_mpool_openib_register: ibv_reg_mr(0x2a96641000,1060864)
>>failed with error: Cannot allocate memory
> 
> 
> .. which means you need to raise that memlock limit higher than your
> current 8192, which is only 8 megabytes. Most MPIs on IB lock lots of
> memory, more if you use more processes, and more for apps that send
> large messages.
> 
> -- greg
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf