[Beowulf] TOE on Linux?

[Joe Landman]

Mark Hahn wrote:
>> application environments.  Contrary to the detractors of the 
>> technologies comments, the TOE/RDMA card *did* provide fairly 
>> significant performance delta for real apps running MPI over gigabit 
>> ethernet.
> 
> well, let's not conflate these things: TOE and RDMA are not intrinsically
> related...

Agreed, but you had both on this card, and you couldn't turn one off ...

> 
>> I won't talk on the business side of them.  We did see 4x better 
>> (wallclock) time on customers StarCD calculations being run over the 
>> TOE/RDMA engine than over the pure gigabit path.
> 
> do you have a sense for whether this was due to RDMA giving you lower 
> latency
> by bypassing the stack (either host or the nic-toe stack)?  I don't think
> I've ever heard of TOE itself providing any latency benefit.

Usually a detriment to latency, but as I remember, they set up a fast 
path for TCP offload via an iWarp/RDMA engine.  So it bypassed the stack.

> 
> also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
> rather something like GAMMA or OpenMX.  I don't suppose you had a chance 
> to test those, did you?

Sadly no.  We are playing with OpenMX now though, for a different code. 
  And Ammasso is long gone.

> 
>> Customer was pleased until they realized that the design of this card 
>> effectively killed their maximum memory size (3.1 GB vs the 4 GB 
>> installed).  This was the iWarp bit.
> 
> I'm guessing that techniques from the IB world would ameliorate this...

:)

I personally don't see the value in RDMA for gigabit (10 GbE yes), when 
IB is ~$180/port (HCA + cable) on the client side and ~$50/port on the 
switch side.  The gigabit switch you need for the low latency bits are 
around $100/port, and then the offload/RDMA NIC is > $500/port ....


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615