Overview : Community : Archive : Tools : Showcase

Archives


- Beowulf
- Beowulf Announce
- Scyld-users
- Beowulf on Debian

[Beowulf] Bonding Gbe using tg3 drivers on an HP cluster

Many of your questions may have already been answered in earlier discussions or in the FAQ. The search results page will indicate current discussions as well as past list serves, articles, and papers.

Search

Steffen Persvold sp at scali.com
Mon Nov 14 00:02:16 PST 2005 

Walid,

To achieve your objective you need two switches (or back-to-back cables).
"Trunking" mode on a switch is not the same as "round-robin", i.e it will
not alternate ports when sending data. Trunking mode in a switch usually
implies a hashing algorithm where the sender and receiver MAC address is put
through a hash to find which port to use. Thus, you can't achieve more than
1Gps with two servers only. However, if you had a large network (with a lot
of different MAC addresses), you would get more than 1Gps from one server,
_if_ it was communicating with more than 1 peer at the same time.

Kind regards,

Steffen Persvold
Technical Director Americas
tel. 508-281-7100 x401
fax. 508-281-7171

http://www.scali.com/
Scaling the Linux datacenter
 

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On
> Behalf Of Walid
> Sent: Thursday, November 10, 2005 2:40 PM
> To: beowulf at beowulf.org
> Subject: [Beowulf] Bonding Gbe using tg3 drivers on an HP cluster
> 
> Dear All,
> 
> Reading through the bonding.txt document, I am failing to achive more
> than 1Gbps on a two  bonded  builtin 1 Gbe network interfaces on a
> proliant server DL-380,  what are we doing wrong, please find below
> our configuration
> 
> Walid.
> 
> 
> 
> Here is the test configuration we have
> 
>  +----------+                 +---------------+
>   +--------+
>   |           |eth0  port-E3|               +--------------------------+
> host2
>   |  host1 +-------------+  switch    | port-E4                    |
>   |          +-----trk13-----+  Procurve |              trk14       |
>   | RHEL 3.0 |eth1  port-F3|   5308 +--------------------------+ RHEL3.0
>  +----------+                  +-----------+ port-F4                  +---
> -----+
> 
> Objective:
>       Aggregate the bandwidth to achieve a 2 GB link, and active fail
> over capabilities
> 
> Requirements:
>       Configure the bonding from both sides of the switch, and the
> host to achieve the above objectives without causing network
> degradation or instability.
> 
> Configuration :
> 
> As above in the diagram
> The Operating system is RHEL3 Update 2, the kernel currently is
> 2.4.21-37.EL from Update 5 on both machines host1, and host2
> 
> The trunking option used in the switch was fec to trunk E3,F3 that are
> connected to host1, and E4, and F4 to host2
> 
> /etc/modules.conf have the following :
> 
> alias eth0 tg3
> alias eth1 tg3
> alias scsi_hostadapter cciss
> alias usb-controller usb-uhci
> alias usb-controller1 ehci-hcd
> alias bond0 bonding
> 
>  [root at xrdbm3 root]# cat /proc/net/bonding/bond0
> 
> Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
> Bonding Mode: load balancing (round-robin)
> MII Status: up
> MII Polling Interval (ms): 0
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> 
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1f
> 
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1e
> Ifconfig -a gives the following :
> bond0     Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
> 
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:782876 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:78540 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:64669079 (61.6 Mb)  TX bytes:12222551 (11.6 Mb)
> 
> eth0      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:120113 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:39272 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:22234287 (21.2 Mb)  TX bytes:6111991 (5.8 Mb)
>           Interrupt:25
> 
>  eth1      Link encap:Ethernet  HWaddr 00:14:38:C3:EE:1F
>           inet addr:10.8.194.203  Bcast:10.8.194.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:662763 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:39268 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:42434792 (40.4 Mb)  TX bytes:6110560 (5.8 Mb)
>           Interrupt:26
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:2666075 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2666075 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
> 
>           RX bytes:191362627 (182.4 Mb)  TX bytes:191362627 (182.4 Mb)
> 
> And ethtool gives the following
> 
> [root at xrdbm3 root]# ethtool eth0
> Settings for eth0:
>         Supported ports: [ MII ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 1
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: g
>         Wake-on: d
> 
>         Current message level: 0x000000ff (255)
>         Link detected: yes
> [root at xrdbm3 root]# ethtool eth1
> Settings for eth1:
>         Supported ports: [ MII ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 1
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: g
>         Wake-on: d
>         Current message level: 0x000000ff (255)
>         Link detected: yes
> --------------------------------------------------------------------------
> -------------------------------------
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network
> 
> NETWORKING=yes
> HOSTNAME=xrdbm3
> GATEWAYDEV=bond0
> GATEWAY=10.8.194.250
> 
> cat /etc/sysconfig/network-scripts/ifcfg-eth0
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
> 
> DEVICE=eth1
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
> 
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
> 
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> IPADDR=10.8.194.203
> NETMASK=255.255.255.0
> NETWORK=10.8.194.0
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list

Home | Overview | Community | Archive | Tools | Showcase

Copyright © 2004 Beowulf.org. All Rights Reserved.