howto increase MTU size on 100Mbps FE

[Josip Loncaric]

Donald Becker wrote:
> 
> On Fri, 26 Apr 2002, Joseph Mack wrote:
> 
> > I know that jumbo frames increase throughput rate on GigE and was
> > wondering if a similar thing is possible with regular FE.
> 
> I used to track which FE NICs support oversized frames.  Jumbo frames
> turned out to be so problematic that I've stopped maintaining the table.
> [...]

> The backwards compatibility issue is severe.

Jumbo frames are great to reduce host frame procesing overhead, but,
unfortunately, we arrived at the same conclusion: jumbo frames and
normal equipment do not mix well.  If you have a separate network where
all participants use jumbo frames, fine; otherwise, things get messy.

Alteon (a key proponent of jumbo frames) has some suggestions: define a
normal frame VLAN including everybody and a (smaller) jumbo frame VLAN;
then use their ACEswitch 180 to automatically fragment UDP datagrams
when routing from a jumbo frame VLAN to a non-jumbo frame VLAN (TCP is
supposed to negotiate MTU for each connection, so it should not need
this help).  This sounds simple, but it requires support for 802.1Q VLAN
tagging in Linux kernel if a machine is to participate in both jumbo
frame and in non-jumbo frame VLAN.  Moreover, in practice this mix is
fragile for many reasons, as Donald Becker has explained...

One of the problems I've seen involves UDP packets generated by NFS. 
When a large UDP packet (jumbo frame MTU=9000) is fragmented into 6
standard (MTU=1500) UDP packets, the receiver is likely to drop some of
these 6 fragments because they are arriving too closely spaced in time. 
If even one fragment is dropped, the NFS has to resend that jumbo UDP
packet, and the process can repeat.  This results in a drastic NFS
performance drop (almost 100:1 in our experience).  To restore
performance, you need significant interrupt mitigation on the receiver's
NIC (e.g. receive all 6 packets before interrupting), but this can hurt
MPI application performance.  NFS-over-TCP may be another good solution
(untested!).

We got good gigabit ethernet bandwidth using jumbo frames (about 2-3
times better than normal frames using NICs with Alteon chipsets and the
acenic driver), but in the end full compatibility with existing
non-jumbo equipment won the argument: we went back to normal frames. 
The frame processing overhead does not seem as bad now that CPUs are so
much faster (2GHz+), even with our gigabit ethernet, and particularly
not with fast ethernet.

However, if we had a separate jumbo-frame-only gigabit ethernet network,
we'd stick to jumbo frames.  Jumbo frames are simply a better solution
for bulk data transfer, even with fast CPUs.

Sincerely,
Josip

-- 
Dr. Josip Loncaric, Research Fellow               mailto:josip at icase.edu
ICASE, Mail Stop 132C           PGP key at http://www.icase.edu./~josip/
NASA Langley Research Center             mailto:j.loncaric at larc.nasa.gov
Hampton, VA 23681-2199, USA    Tel. +1 757 864-2192  Fax +1 757 864-6134