Beowulf Questions

[Mark Hahn]

> > there simply is no coming breakthrough that will make all networking
> > fast, low-latency, cheap, ubiquitous and low-power.  and grid
> > (in the grand sense) really does require *all* those properties.
> 
> I'm not quite sure. The only hard limit on latency is relativistic (in
> vacuum, 1 ns = 0.3 m; 10 ns = 3 m, 100 ns = 30 m; 1 us = 3 km; 10 us = 30
> km, 100 us = 300 km).

sure, though I admit I was thinking about 1 ns per foot ;)

I think the appeal of Grids is to farm really large collections of 
underutilized processors.  to me, that means more than just a couple
of buildings worth (which would be in ~5 us diameter - a typical 
latency for serious non-loosely-coupled clustering today.)

> Right now, commercial networks based on GBit fiber
> Ethernet backbones exist, delivering sub-ms latency to end consumers. 

1 ms!  jeez, I think that stretches the definition of clustering,
even the incredibly loose-coupled end of the field that people call Grid.

> 10 GBit fiber Ethernet will be starting to displace GBit Ethernet in that
> niche.

a nice bandwidth improvement.  no help for latency, and really, the existence
of a 10Gb campus backbone might not change real app-level performance much.
(the mere existence of 10Gb will generate competing consumers.  for instance,
I expect any university with a 10Gb backbone already does VOIP and possibly
video over it, not to mention security card scanners, centralized file/backup
services, more undergrad web surfing...)

> At 10 GBps fiber acts as a FIFO, containing ~50 bit/m (50 kBit/km)
> of fiber allowing (admittedly, there is no impetus for developing
> cut-through WAN transmission technology) almost purely photonically
> switched networks where routing latency is negligible in regards to
> relativistic latency. That assumes that the fiber(s) is unloaded, of
> course, as store-and forward will suddenly result in lousy latency. This 
> can't happen on a true crossbar-switched LAN.

it's neat stuff.  being a pessimist, I look at "street-level" bandwidth.
I guesstimate that over internet WANs in a typical non-podunk university,
I've see 2-4x improvement in bandwidth versus 5 years ago.  no real change 
in latency, since it's geographic.  I don't know about you, but my DSL
at home, considered a pretty good service, is capped at 100 KB/s down
and a measly 15 KB/s up.  that's only moderately better than dialup 
(even ignoring the fact that cable/dsl providers are usually a lot
more hops away than a university's own modem pool, meaning vastly more lossy)

> This clearly can't compete with dedicated ultralocal interconnects like
> Myrinet & Co, but it indicates GBit based clusters need not to be located
> physically close.

for seti at home type stuff, which is extremely latency-tolerant.  of course,
this is not news at all, since mainframes have been sharing files over 
~10 mile distances for a long time.  I still don't see that much has changed:
smallish improvements (10X or less in 5 years) and somewhat cheaper.

I think the main point is that networking is improving much more slowly
than many other metrics related to computers.  though as usual, if you
only compare network latency to disk and dram latency, they all look
pretty similarly flat.  

but that flatness was my original point: Grid, if it is to break out of the 
seti at home ghetto, must assume that networking will improve dramatically.
that doesn't seem to be happening, for physical, practical, political
and economic reasons...