[Beowulf] fast interconnects

[Mark Hahn]

> > 3) Then, there's the switch.  If it's a *switch* that sets up circuits, 
> > then it can be fairly inexpensive.  If it's something that has to do packet 
> > buffering, it gets a lot more expensive.
> 
> Switches (crossbars) don't scale for very many ports. Especially if you have to

I'm a little mystified by this comment, since you can build arbitrary
topologies using xbar primitives.  for instance, quadrics fabrics are 
usually full-bisection fat-trees, and are composed of (iirc) 8x xbar
switching chips.  myrinet is <handwave>similar</handwave>.

> do cut-through switching at those high speeds. It would be good if each NIC
> would came with an integrated switch, with enough ports to wire at least a 3d
> torus (where you route/switch messages via Bresenham).

while I like multi-port designs, they do become huge investments in wire.

> In regards to keeping the wires short, does this IBM trick of keeping all
> wires equal-length work well on 3d lattices, and above? This would seem to
> be a must for those coming (hopefully) Hypertransport motherboards with 
> connectors.

afaikt, everyone now takes the approach of putting eq/deskew/etc logic
on every pair or smallnumber of wires.  HT certainly does that (including
a clock per 8 data bits).  I think pci-e is based on separate analog
processing for each lane.