[Beowulf] Bonding Gbe using tg3 drivers on an HP cluster
Steffen Persvold
sp at scali.com
Mon Nov 14 00:02:16 PST 2005
Walid,
To achieve your objective you need two switches (or back-to-back cables).
"Trunking" mode on a switch is not the same as "round-robin", i.e it will
not alternate ports when sending data. Trunking mode in a switch usually
implies a hashing algorithm where the sender and receiver MAC address is put
through a hash to find which port to use. Thus, you can't achieve more than
1Gps with two servers only. However, if you had a large network (with a lot
of different MAC addresses), you would get more than 1Gps from one server,
_if_ it was communicating with more than 1 peer at the same time.
Kind regards,
Steffen Persvold
Technical Director Americas
tel. 508-281-7100 x401
fax. 508-281-7171
http://www.scali.com/
Scaling the Linux datacenter
> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On
> Behalf Of Walid
> Sent: Thursday, November 10, 2005 2:40 PM
> To: beowulf at beowulf.org
> Subject: [Beowulf] Bonding Gbe using tg3 drivers on an HP cluster
>
> Dear All,
>
> Reading through the bonding.txt document, I am failing to achive more
> than 1Gbps on a two bonded builtin 1 Gbe network interfaces on a
> proliant server DL-380, what are we doing wrong, please find below
> our configuration
>
> Walid.
>
>
>
> Here is the test configuration we have
>
> +----------+ +---------------+
> +--------+
> | |eth0 port-E3| +--------------------------+
> host2
> | host1 +-------------+ switch | port-E4 |
> | +-----trk13-----+ Procurve | trk14 |
> | RHEL 3.0 |eth1 port-F3| 5308 +--------------------------+ RHEL3.0
> +----------+ +-----------+ port-F4 +---
> -----+
>
> Objective:
> Aggregate the bandwidth to achieve a 2 GB link, and active fail
> over capabilities
>
> Requirements:
> Configure the bonding from both sides of the switch, and the
> host to achieve the above objectives without causing network
> degradation or instability.
>
> Configuration :
>
> As above in the diagram
> The Operating system is RHEL3 Update 2, the kernel currently is
> 2.4.21-37.EL from Update 5 on both machines host1, and host2
>
> The trunking option used in the switch was fec to trunk E3,F3 that are
> connected to host1, and E4, and F4 to host2
>
> /etc/modules.conf have the following :
>
> alias eth0 tg3
> alias eth1 tg3
> alias scsi_hostadapter cciss
> alias usb-controller usb-uhci
> alias usb-controller1 ehci-hcd
> alias bond0 bonding
>
> [root at xrdbm3 root]# cat /proc/net/bonding/bond0
>
> Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
> Bonding Mode: load balancing (round-robin)
> MII Status: up
> MII Polling Interval (ms): 0
> Up Delay (ms): 0
> Down Delay (ms): 0
>
>
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1f
>
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 0
> Permanent HW addr: 00:14:38:c3:ee:1e
> Ifconfig -a gives the following :
> bond0 Link encap:Ethernet HWaddr 00:14:38:C3:EE:1F
>
> inet addr:10.8.194.203 Bcast:10.8.194.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
> RX packets:782876 errors:0 dropped:0 overruns:0 frame:0
> TX packets:78540 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:64669079 (61.6 Mb) TX bytes:12222551 (11.6 Mb)
>
> eth0 Link encap:Ethernet HWaddr 00:14:38:C3:EE:1F
> inet addr:10.8.194.203 Bcast:10.8.194.255 Mask:255.255.255.0
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:120113 errors:0 dropped:0 overruns:0 frame:0
> TX packets:39272 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:22234287 (21.2 Mb) TX bytes:6111991 (5.8 Mb)
> Interrupt:25
>
> eth1 Link encap:Ethernet HWaddr 00:14:38:C3:EE:1F
> inet addr:10.8.194.203 Bcast:10.8.194.255 Mask:255.255.255.0
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
> RX packets:662763 errors:0 dropped:0 overruns:0 frame:0
> TX packets:39268 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:42434792 (40.4 Mb) TX bytes:6110560 (5.8 Mb)
> Interrupt:26
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:2666075 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2666075 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
>
> RX bytes:191362627 (182.4 Mb) TX bytes:191362627 (182.4 Mb)
>
> And ethtool gives the following
>
> [root at xrdbm3 root]# ethtool eth0
> Settings for eth0:
> Supported ports: [ MII ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 1
> Transceiver: internal
> Auto-negotiation: on
> Supports Wake-on: g
> Wake-on: d
>
> Current message level: 0x000000ff (255)
> Link detected: yes
> [root at xrdbm3 root]# ethtool eth1
> Settings for eth1:
> Supported ports: [ MII ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Half 1000baseT/Full
> Advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 1
> Transceiver: internal
> Auto-negotiation: on
> Supports Wake-on: g
> Wake-on: d
> Current message level: 0x000000ff (255)
> Link detected: yes
> --------------------------------------------------------------------------
> -------------------------------------
>
> [root at xrdbm3 root]# cat /etc/sysconfig/network
>
> NETWORKING=yes
> HOSTNAME=xrdbm3
> GATEWAYDEV=bond0
> GATEWAY=10.8.194.250
>
> cat /etc/sysconfig/network-scripts/ifcfg-eth0
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
>
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
>
> DEVICE=eth1
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> MASTER=bond0
> SLAVE=yes
>
> [root at xrdbm3 root]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
>
> DEVICE=bond0
> BOOTPROTO=none
> ONBOOT=yes
> USERCTL=no
> IPADDR=10.8.194.203
> NETMASK=255.255.255.0
> NETWORK=10.8.194.0
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf
mailing list