[Beowulf] interconnect and compiler ?

[Greg Lindahl]

On Fri, Jan 30, 2009 at 09:24:11PM +0100, Vincent Diepeveen wrote:

> For example: If one of the cores c0 on our box is busy receiving a
> long message from a remote node in the network, a message that will
> take significant time, can it switch in between let go through a
> short message meant for c1, and if so what latency time does it take
> to receive it for c1?

Most modern networks are packet switched. So if you're receiving a
many-packet message from one sender, frequently a short message from
another sender will be able to arrive in the middle. There are a fair
number of details to it, though: there are queues in the switch chips,
and the short message might have to wait for a queue time at every
switch chip.

Because this could be a long time, most modern networks have multiple
priority levels. In InfiniBand this is called a VL, a virtual lane. I
don't know anyone who's used them with a good result, and I suspect
that in practice using multiple VLs will suffer from significant
negative effects due to implementation details. Does anyone know of a
proof point of this?

-- greg