[Beowulf] 1.2 us IB latency?

[greg.lindahl at qlogic.com]

On Wed, Apr 18, 2007 at 03:59:06PM -0400, Steffen Persvold wrote:

> I'm just curious what is believed to be "small messages" in this context
> today ?

Oh, in this case I was meaning "up to 64 kbytes". Our current break
point between PIO and DMA is 64000 bytes.

> Back then we were struggling with PIO transfers and how they were
> treated in the CPU/North bridge (write combining and all that). I
> believe this might still be an issue, correct ?

Yes and no. We haven't run into any hardware issues with modern
hardware; there have been some quirks, but we have overcome them.

>From the software point of view, manipulating MTRRs has proven to be
problemmatic with a few BIOSes that don't have a "discrete mtrr"
option. Apparently these days few or no graphics cards currently use
write combining, so it's only a few network cards that do. In most
cases we can manipulate the mtrrs after boot to fix this. Getting
formal support for PAT in the Linux kernel is the long-term fix for
this.

-- greg