Beowulf: A theorical approach

[Jakob Østergaard]

On Thu, 22 Jun 2000, Robert G. Brown wrote:

> On Thu, 22 Jun 2000, [iso-8859-1] Jakob Østergaard wrote:
> 
> > On Thu, 22 Jun 2000, Greg Lindahl wrote:
> > 
> > > > Just as a matter of curiosity -- Once upon a time some two or three
> > > > years ago I suggested on the list that a development company consider
> > > > building a network communications device that plugged into the second
> > > > CPU slot of a dual CPU board.
> > > 
> > > It's hard to build something that plugs into an Intel-designed CPU bus.
> > > Scali tried it and it didn't work so hot.
> > 
> > At least on Intel hardware, SMP systems have cache coherency in the hardware,
> > and that's _expensive_ wrt. inter-cpu communication.  There's write snooping
> > and all sorts of the strangest things happening.  I cannot imagine how this
> > could work in any way over a network with any reasonable performance. Even
> > given infinite bandwidth, I guess the latency from ten meters of copper wiring
> > would kill performance.  (any sub-c interconnect would and tunneling is a few
> > years away I guess  ;)
> 
> Not arguing with the difficulty of doing the local engineering, but from
> whence the remarks concerning copper and latency?  It could be an
> optical interconnect, or a three meter copper interconnect for all I
> care.  Even ten meters is only 33 nanoseconds at c which is a WHOLE lot
> smaller than the other sources of latency in a normal network
> interconnect. I thought signals propagated on copper at an appreciable
> fraction of c... am I missing something here?

I don't know exactly how often the CPUs go ``what are you doing ?  nothing ?
oh, keep up the good work!'' in an SMP configuration to maintain cache
coherency.  But with 33ns latency you can only do that 15 million times a
second.  While that might seem like a lot, it's not if the CPUs need to do that
every time a new cache line is accessed.   I don't know if that's exactly the
case, and I'd appreciate if anyone in the know could comment on that.

> Never mind.  I'll just go get the beer Greg suggested instead;-)

Right on !   In any case, this little discussion is sort of moot as the CPU
interconnect was for message passing and not cache coherency anyway.       :)

-- 
................................................................
: jakob at ostenfeld.dtu.dk  : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............: