[Beowulf] TOE on Linux?

[Joe Landman]

Prentice Bisbal wrote:
> Prentice Bisbal wrote:
>> Does anyone know of any network cards/drivers that support TOE (TCP
>> Offload Engine) for Linux? A hardware vendor just told me that Linux
>> does not support the TOE features of *any* network card.
>>
>> Given Linux's strong presence in HPC and the value of having TOE in a
>> cluster, I find that hard to believe.
>>
> 
> I should have googled before asking that question. I googled for "Linux
> TCP Offload Engine" and got an eyeful. Apparently TOE = Evil to Linux
> developers.

Ok... worth noting that there are opinions and many people have them. 
We have had experience deploying and using Ammasso RDMA/TOE cards in 
real world application environments.  Contrary to the detractors of the 
technologies comments, the TOE/RDMA card *did* provide fairly 
significant performance delta for real apps running MPI over gigabit 
ethernet.

I won't talk on the business side of them.  We did see 4x better 
(wallclock) time on customers StarCD calculations being run over the 
TOE/RDMA engine than over the pure gigabit path.  Customer was pleased 
until they realized that the design of this card effectively killed 
their maximum memory size (3.1 GB vs the 4 GB installed).  This was the 
iWarp bit.

Ok, now that I have said that (and expect quite a few comments about how 
it really doesn't work :( and the Linux /usr/bin/time command must have 
been broken that day ... ), Chelsio and a number of others do have 
TOE/RDMA cards.  Their drivers are in the OFED stack.  Be prepared to 
pay quite a bit for these cards.  Before you do, have a good hard look 
at Infiniband with RDMA capability.  Does your app really need TCP 
offload, or simply fast path MPI processing?  Myricom, Mellanox, and 
others can provide this through the respective stacks.

Basically I am not arguing against TOE, just I don't see the need in 
many cases today.  This is not to say it doesn't work.  It does.  It 
just costs a bit to get it working, and you have to do the cost-benefit 
analysis to determine whether or not you really need it, especially 
compared to other lower cost technologies.

Put another way, what problem are you trying to solve where you believe 
TOE to be the right answer?  There may be alternative answers that have 
less pain associated with them.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615