[Beowulf] Pony: not yours.
hahn at mcmaster.ca
Thu May 16 09:50:17 PDT 2013
>> that since we exist in 3d, our networks need to be a 3d lattice at scale.
>> fighting power (flops, comm) seems like a noble, *engineering* fight,
>> but fighting our existence's dimensionality is silly...
> We already exist in 3d - racks. We limit rack heights due to weight already
> (another item to add to your list of limits above).
weight (and 2d footprint) are purely linear, and not very interesting,
to me at least. it's demonstrably possible (not hard) to do racks
in 3d - and in fact doing so generally draws more attention to the
macro-structure of the cluster anyway (including airflow, for instance.)
> I understand that you are arguing for 3d is silicon (or other substrate),
> but there will still be a practical limit.
I think I'm actually making the opposite argument. that at least for
inter-unit networking, the topology for very scaled clusters needs to be
a 2 or 3d mesh. if you're building a cluster laid out in 2d, as most are,
then there are scaling problems with any non-2d topology. I love higher-
order fabrics (fat trees, hypercubes, dragonfly, kautz, etc), but they
have a sweet spot in their scale, and the whole point of exascale seems
to be to exceed all sweet spots.
within a "unit" (chassis, rack, take your pick), it's practical to exceed
the "consensual dimensionality" - especially since you get to use different
materials (say, a backplane with a complex pattern of connections
implementing a 5d torus, etc). inside the package, the coming win seems
to be simply 2.5d integration to minimize compute-memory distances,
though this is nothing new to the IBM MCM world.
> I saw some presentation a year or two ago that showed an exascale system as
> a sphere with the switch complex in the center. It looked very 60s-ish,
works for me! though if you're going to do that, it would be awefully
tempting to try to do free-space optics. but again, once you exceed
the sweetspot for your free-space-optical-sphere-cluster, you have to
pack multiple spheres, and that's going to be some kind of mesh...
>> yes, I would like to demo your new blackhole-based interconnect! ;)
> Shouldn't it be wormhole-based? The blackhole-based one will just destroy
> the data, no?
I knew I should have just said "singularity" there!
More information about the Beowulf