[Beowulf] NFS over RDMA performance confusion

[holway at th.physik.uni-frankfurt.de]

> On 09/13/2012 07:52 AM, holway at th.physik.uni-frankfurt.de wrote:
>
> [...]
>
>> If I set up a single machine to hammer the fileserver with IOzone I see
>> something like 50,000 IOPS but if all four machines are hammering the
>> filesystem concurrently we got it up to 180,000 IOPS.
>
> I wouldn't recommend IOzone for this sort of testing.  Its not a very
> good load generator, and it has a tendency to report things which are
> not actually seen at the hardware level.  I'd noticed this some years
> ago, when running some of our benchmark testing on these units, that an
> entire IOzone benchmark completed with very few activity lights going on
> the disks.  Which suggested that the test was happily entirely cached,
> and I was running completely within cache.

Hmm, It has thus far behaved in a very predictable manner. I am always
using it in o_direct mode with the -I flag and/or -o o_sync mode. It has
always given me results consistant with dd.

>
> Use fio.

I will take a look.

>
> Second, are the disks behind the NFS/ZFS server solid state, ram disk,
> or spinning rust?

I have 20x 15k sas drives in 10 mirrored sets. An STEC IOPS SSD as read
cache and an 8GB STEC ZEUS battery backed RAM device for my SLOG/ZIL.

Also lots and lots of mem.

>
> Are you asking why a single machine cannot fill your QDR bandwidth?

That is exactly my question.

>
> I'd recommend running traces on the individual machines to see where
> things are getting lost.  One you have the traces, post em, and see if
> people can help.

https://th.physik.uni-frankfurt.de/~holway/strace-trunkd.log
https://th.physik.uni-frankfurt.de/~holway/strace.log

I have chopped out all the data from the strace in strace-trunkd.log

Not sure this output is completely useful :)

Thanks,

Andrew





>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>         http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>