[Beowulf] Q: IB message rate & large core counts (per node) ?

[Gilad Shainer]

> 
> > BTW, is Cray SeaStar2+ better than IB - for nodes w/many cores ?
> 
> BW seems like a SeaStar2+ advantage, though I haven't seen any
> discussion of latency or message rate.
> 

SeaStar2+ BW to the node is 6.5GB bi-dir, similar to IB QDR 4x port BW
through PCIe Gen2. SeaStar BW to the network is 9.6GB bi-dir, little bit
higher than the 8GB IB BW between switches, but less than IB 12x port
which deliver 24GB/s. I have not see latency numbers out there as well,
but some indications showed that IB latency is lower than SeaStar.


> > And I didn't see latencies comparison for SeaStar vs IB.
> 
> my guess is that for short routes, seastar is competitive,
> but can fall behind on large machines (long routes).
> 
> regardless of how tight the seastar per-hop latency is,
> IB has 2.5x the per-hop fanout (2or 3 outgoing 9.6 GB links
> versus 18 outgoing 4 GB/s links).  higher radix means an
> advantage that increases with size.