[Beowulf] Small files

[Bernd Schubert]

On 06/12/2014 03:09 PM, Jeffrey Layton wrote:
> Tom,
>
> Without digging into the details too much, can you describe the Lustre
> setup? As Bernd alluded to, it might be something in the configuration or
> version that is hampering better performance. But then again, you may not
> want to upgrade to a newer because of disruption. But maybe there are some
> things that can be done to improve Lustre performance.

The problem with millions of files are the hash directories Lustre is 
creating files in - it is a hard job for the kernel to cash the random 
access there.

Here's a (not accepted) patch to improve this issue:

http://lists.openwall.net/linux-ext4/2011/06/20/31

Some patches landed in kernel version 3.11, that improved the situation 
a bit if you set low /proc/sys/vm/vfs_cache_pressure values
(see http://lwn.net/Articles/551016/ ).
But then if you have a very high number of files, there is simply no way 
to cache dentries/inodes properly. That is way we decided to re-work our 
entire storage layout in FhGFS and to avoid hash directories at all.

I don't claim that Tom has suffers from that issue, but with a high 
number of files that is just a common issue.


Cheers,
Bernd