[Beowulf] zfs tuning for HJPC/cluster workloads?

[Michael Brown]

Joe Landman wrote:
> Hi folks:
>
>   Investigating zfs on a Solaris 10 5/08 loaded JackRabbit for a customer. 
> zfs performance isn't that good relative to Linux on this same hardware 
> (literally a reboot between the two environments)
[...]
>   This is for an IO intensive application with multiple threads doing 
> 1-100 GB streaming reads.

Here's your problem. One of the problems with ZFS is it's performace for 
streaming workloads (reads or writes). It's designed for, and does much 
better at, large quantities of smallish unrelated I/O operations. I even 
vaugely remember something in the Solaris documentation recommending you use 
UFS if you're going to be streaming a lot. The only suggestion I have for 
improving ZFS performance in this case is to turn off the prefetching if you 
haven't already. Also, streaming the files on to the disk one at a time does 
help a bit.

Also, regarding ZFS on FreeBSD, from what I've read I wouldn't recommend it 
on a production server just yet. While it has improved dramatically over the 
last year (I haven't seen any recent reports of filesystems being eaten, for 
example), it still has the tendency to crash or lock up machines under 
certain workloads.

-- 
Michael Brown
Add michael@ to emboss.co.nz ---+--- My inbox is always open