[Beowulf] Q: IB message rate & large core counts (per node)?

[Greg Lindahl]

On Tue, Feb 23, 2010 at 05:35:41PM -0500, Brian Dobbins wrote:

>   Well, clearly we hope to move more towards hybrid methods -all that's old
> is new again?-

If you want bad performance, sure. If you want good performance, you
want a device which supports talking to a lot of cores, and then
multiple devices per node, before you go hybrid. The first two don't
require changing your code. The last does.

The main reason to use hybrid is if there isn't enough parallelism in
your code/dataset to use the cores independently.

> But getting back to a technical vein, is the multiplexing an issue due to
> atomic locks on mapped memory pages?  Or just because each copy reserves its
> own independent buffers?  What are the critical issues?

It's all implementation-dependent. A card might have an on-board
memory limit, or a limited number of "engines" which process
messages. Even if it has a option to store some data in main memory,
often that results in a scalability hit.

-- greg