[Beowulf] Re: torus versus (fat) tree topologies

[Rick Friedman]

> simply because the are not enough users of it.  As a consequence, I am
> looking to find the "best" interconnect solution which will allow a
> few people use of most or all of the CPUs for the jobs we run.

	there's always a danger of over-benchmarking, but you should
probably see if you can get access to an IB cluster.  for CFM, I'm a
little surprised you appear to care so much about latency, since I'd
expect your workload to have the usual volume/surface-area scaling, and
thus doing a lot of work in a single node, and needing only moderate
bursts of bandwidth for nontrivial problem sizes.
	
	from looking at list prices on the web, Myrinet, IB and
Dolphinics have similar per-port prices which are noticably lower than
Quadrics but also dramatically higher than gigabit.  I suspect most
people would agree that Quadics is a latency specialist, at least for
not purely nearest-neighbor applications.  OTOH, for cheap nodes, you
should probably consider whether spending 50% of the node price makes
sense for the performance boost.  (I see 242-based servers starting at
around $2k list, and your total gigabit cost would be less than
$100/port.)

Just to add to this list of options, there are currently a number of
companies working on a newer Ethernet technology called RDMA over
TCP/IP. Adapter cards that support this technology delivers lower
latency and higher CPU utilization than standard GigE, but runs on
standard Ethernet infrastructure (switches, cables, etc). 

I admit to working for one of those companies (Ammasso), but wanted to
let you know about other alternatives.

Rick