[Beowulf] tg3 or bcm5700? Options?

[tegner at nada.kth.se]

Hi all,

We have some trouble with the parallel performance of our cluster of dual
opterons MSI-9245 in IBM E325. We are using one of the internal Broadcom
nics (eth0) as a "nfs-network", and the other (eth1) as a "computational
network".

We have tried both the tg3-driver and Broadcoms bcm5700. The tg3-driver
seems to deliver good network performance over eth1 (we have used netpipe
to test). The bcm5700-driver gives higher latency, but on "speedup" tests
(i.e. checking the speedup of a cfd problem of fixed size on different
numbers of processors) the bcm5700-driver gives significantly better
results.

By loading the bcm5700-driver with the options

options bcm5700 adaptive_coalesce=0,0 rx_coalesce_ticks=1,1 \
rx_max_coalesce_frames=1,1 tx_coalesce_ticks=1,1 \
tx_max_coalesce_frames=1,1

the latency is improved, BUT the "speedup" performance is somewhat degraded.

Question is if anyone of you have experienced these kinds of issues, and
if you have suggestions on how the network performance can be "optimized"
(e.g. what options to use in modprobe.conf).

Thanks in advance,

/jon