[Beowulf] Re: Re: Home beowulf - NIC latencies

[Ashley Pittman]

On 14 Feb 2005, at 18:58, Greg Lindahl wrote:
> On Mon, Feb 14, 2005 at 05:42:42PM +0000, Ashley Pittman wrote:
>> If you had a bunch of sends to do to N remote processes then I'd 
>> expect
>> you to post them in order (non-blocking) and wait for them all at the
>> end, the time taken to do this should be (base_latency + ( (N-1) * M 
>> ))
>> where M is the recpipiocal of the "issue rate".  You can clearly see
>> here that even for small number of batched sends (even a 2d/3d nearest
>> neighbour matrix) the issue rate (that is how little CPU the send call
>> consumes) is at least as important that the raw latency.
>
> Unless I completely misunderstand your formula, M is not only the CPU
> the send call consumes. It's easy to find situations (fast cpu, slow
> network) where the cpu consumed isn't a part of M at all. Even for a
> modern 1 GByte/sec network, cpu consumed might not be a part of M.

I'm talking about our (Quadrics) network here which has a CPU offload, 
a Wait call is simply a few function calls, a memory read (to test 
completion),  a mutex lock/unlock cycle and a linked list insertion, 
nothing more.  Some CPU is used in the send call as I said but outside 
the two calls there is zero CPU usage although potentially reduced 
memory to CPU bandwidth.

> Reducing CPU consumed can't hurt. But reasoning about it seems to be
> less useful than testing actual applications.

I do that as well.

Ashley,