[Beowulf] RoCE vs. InfiniBand

[Chris Samuel]

On Thursday, 26 November 2020 3:14:05 AM PST Jörg Saßmannshausen wrote:

> Now, traditionally I would say that we are going for InfiniBand. However,
> for reasons I don't want to go into right now, our existing file storage
> (Lustre) will be in a different location. Thus, we decided to go for RoCE
> for the file storage and InfiniBand for the HPC applications.

I think John hinted at this, but is there a reason for not going for IB for 
the cluster and then using Lnet routers to connect out to the Lustre storage 
via ethernet (with RoCE) ?

https://wiki.lustre.org/LNet_Router_Config_Guide

We use Lnet routers on our Cray system to bridge between the Aries 
interconnect inside the XC to the IB fabric our Lustre storage sits on.

All the best,
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA