[Beowulf] fast interconnects
hahn at physics.mcmaster.ca
Tue May 23 07:52:18 PDT 2006
> > 3) Then, there's the switch. If it's a *switch* that sets up circuits,
> > then it can be fairly inexpensive. If it's something that has to do packet
> > buffering, it gets a lot more expensive.
> Switches (crossbars) don't scale for very many ports. Especially if you have to
I'm a little mystified by this comment, since you can build arbitrary
topologies using xbar primitives. for instance, quadrics fabrics are
usually full-bisection fat-trees, and are composed of (iirc) 8x xbar
switching chips. myrinet is <handwave>similar</handwave>.
> do cut-through switching at those high speeds. It would be good if each NIC
> would came with an integrated switch, with enough ports to wire at least a 3d
> torus (where you route/switch messages via Bresenham).
while I like multi-port designs, they do become huge investments in wire.
> In regards to keeping the wires short, does this IBM trick of keeping all
> wires equal-length work well on 3d lattices, and above? This would seem to
> be a must for those coming (hopefully) Hypertransport motherboards with
afaikt, everyone now takes the approach of putting eq/deskew/etc logic
on every pair or smallnumber of wires. HT certainly does that (including
a clock per 8 data bits). I think pci-e is based on separate analog
processing for each lane.
More information about the Beowulf