[Beowulf] Network Filesystems performance

[Bogdan Costescu]

On Thu, 23 Aug 2007, Glen Dosey wrote:

> What really gets me is that while my NFS reads are around ~50MB/s , 
> the writes are basically at wire speed, slowing down to and holding 
> at about ~90MB/s when we exceed the 4GB file size. That would seem 
> to indicate to me the server has no problem dealing with a saturated 
> NIC and reasonably high I/O on the QLA2342 at the same time.

I beg to disagree with your conclusion. At least for networking the Tx 
and Rx paths are quite different in what operations they do and 
especially how many interrupts they need (due to possible interrupt 
mitigation, in either driver or hardware); I think that for the SCSI 
stack used to talk to a FC controller the situation is quite similar. 
So, I think that a test that would better aproximate the NFS read case 
would be:
- run the local reading test on the server (using dd)
- run at the same time a network speed test (using f.e. ttcp) which 
sends data from the server to a client

The slowing down that you mention above as happening at 4GB looks to 
me like the server's memory used for caching fills up and the writing 
to disk is then forced; so it's not the 4GB file-size that is changing 
behaviour, but the caching that doesn't help anymore. If you want to 
better test the behaviour, try running on a client bonnie++ with a 
file size of 8GB or more (although the client's memory size could also 
play a role in caching).

>>  50:      69688     963247    1227309     270953   IO-APIC-level  qla2xxx
>>  58:      15112      96722      96347       7613   IO-APIC-level  qla2xxx
>>  66:   47398161          0          0          0   IO-APIC-level  eth0

Seems like the interrupts for the NIC go to only one CPU, while the 
ones for the FC controller are spread among CPUs. I think that memory 
locality effects come into play here and reduce the speed. Did you 
play by any chance with CPU affinity for interrupts ?

>> atop shows basically the same iostat does, which is that on the initial
>> read the FC disk is about %85 percent utilized

This seems strange: for 40MB/s the disk is used at 85%, but then you 
report getting 160MB/s ? It doesn't add up for me - do you have an 
explanation ?

--
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De