[Beowulf] Bonding Gbe using tg3 drivers on an HP cluster

Thu Nov 10 05:40:27 PST 2005

Dear All,

Reading through the bonding.txt document, I am failing to achive more
than 1Gbps on a two  bonded  builtin 1 Gbe network interfaces on a
proliant server DL-380,  what are we doing wrong, please find below
our configuration

Walid.

Here is the test configuration we have

 +----------+                 +---------------+                       
  +--------+
  |           |eth0  port-E3|               +--------------------------+  host2
  |  host1 +-------------+  switch    | port-E4                    |
  |          +-----trk13-----+  Procurve |              trk14       |
  | RHEL 3.0 |eth1  port-F3|   5308 +--------------------------+ RHEL3.0
 +----------+                  +-----------+ port-F4                  +--------+

Objective:
      Aggregate the bandwidth to achieve a 2 GB link, and active fail
over capabilities

Requirements:
      Configure the bonding from both sides of the switch, and the
host to achieve the above objectives without causing network
degradation or instability.

Configuration :

As above in the diagram
The Operating system is RHEL3 Update 2, the kernel currently is
2.4.21-37.EL from Update 5 on both machines host1, and host2

The trunking option used in the switch was fec to trunk E3,F3 that are
connected to host1, and E4, and F4 to host2

/etc/modules.conf have the following :

alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter cciss
alias usb-controller usb-uhci
alias usb-controller1 ehci-hcd
alias bond0 bonding

 [root at xrdbm3 root]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:14:38:c3:ee:1f

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:14:38:c3:ee:1e
Ifconfig –a gives the following :
bond0     Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F

          inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:782876 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78540 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:64669079 (61.6 Mb)  TX bytes:12222551 (11.6 Mb)

eth0      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
          inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:120113 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:22234287 (21.2 Mb)  TX bytes:6111991 (5.8 Mb)
          Interrupt:25

 eth1      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
          inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:662763 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:42434792 (40.4 Mb)  TX bytes:6110560 (5.8 Mb)
          Interrupt:26
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2666075 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2666075 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

          RX bytes:191362627 (182.4 Mb)  TX bytes:191362627 (182.4 Mb)

And ethtool gives the following

[root at xrdbm3 root]# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d

        Current message level: 0x000000ff (255)
        Link detected: yes
[root at xrdbm3 root]# ethtool eth1
Settings for eth1:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes
---------------------------------------------------------------------------------------------------------------

[root at xrdbm3 root]# cat /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=xrdbm3
GATEWAYDEV=bond0
GATEWAY=10.8.194.250

cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes

[root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes

[root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
IPADDR=10.8.194.203
NETMASK=255.255.255.0
NETWORK=10.8.194.0