[Beowulf] ARP timers on RHEL4 vs. RHEL5

[Tom Ammon]

Hi,

I've been trying to figure out how to adjust the ARP timeout on kernel
2.6.9 and I found the following in /proc/sys/net/ipv4/neigh/ib0 (its an
IB interface I am interested in changing) with the following values.
This is on kernel 2.6.9-89ELsmp (RHEL4) :

[root at up255 ib0]# cat anycast_delay
99
[root at up255 ib0]# cat app_solicit
0
[root at up255 ib0]# cat base_reachable_time
30
[root at up255 ib0]# cat delay_first_probe_time
5
[root at up255 ib0]# cat gc_stale_time
60
[root at up255 ib0]# cat locktime
99
[root at up255 ib0]# cat mcast_solicit
3
[root at up255 ib0]# cat proxy_delay
79
[root at up255 ib0]# cat proxy_qlen
64
[root at up255 ib0]# cat retrans_time
99
[root at up255 ib0]# cat ucast_solicit
3
[root at up255 ib0]# cat unres_qlen
3

When I test this, along with per-flow ECMP (using the iproute2 utils), I
see that the ARP cache is timing out about every 10 minutes (I observe
this by load balancing an iperf flow between two different gateway
machines and then graphing the interface traffic)

On a newer kernel, 2.6.18-164.11.1.el5 (RHEL5), I see mostly the same
parms available, but a few new ones have been added. However, all of the
parms that are the same name between the two kernels are the same values:

[root at gateway2 ib0]# cat anycast_delay
99
[root at gateway2 ib0]# cat app_solicit
0
[root at gateway2 ib0]# cat base_reachable_time
30
[root at gateway2 ib0]# cat base_reachable_time_ms
30000
[root at gateway2 ib0]# cat delay_first_probe_time
5
[root at gateway2 ib0]# cat gc_stale_time
60
[root at gateway2 ib0]# cat locktime
99
[root at gateway2 ib0]# cat mcast_solicit
3
[root at gateway2 ib0]# cat proxy_delay
79
[root at gateway2 ib0]# cat proxy_qlen
64
[root at gateway2 ib0]# cat retrans_time
99
[root at gateway2 ib0]# cat retrans_time_ms
1000
[root at gateway2 ib0]# cat ucast_solicit
3
[root at gateway2 ib0]# cat unres_qlen
3

Yet when I observe the same traffic flow with this machine, the ARP
cache times out about once per minute.

Is there another set of parameters somewhere that govern how often the
kernel times out the ARP cache? If so, where might I find that? Is there
any kernel documentation that talks about changing ARP timers on the
linux kernel?

Tom Ammon

-- 
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu