[Beowulf] Small files

Tom Harvill tom.harvill at unl.edu
Wed Jun 11 12:03:28 PDT 2014


This is my first time posting to this list, thanks in advance for any 
time you spend

We've found that a large majority of our files (~40MM of ~50MM) are less 
than 10KB.
We believe our filesystem (lustre) is bottlenecked with IOPs and locking 
related to
jobs running against these files.  We have ~700TB usable storage with 
~500TB consumed,
almost all consumption is by a relatively small number of very very 
large files.

I want to ask this general question: how does your shop deal with the 
general problem of
small files in filesystems on (beowulf) compute clusters? Specifically, 
files that users expect
to actively use for read and write operations for their research.

Do you distinguish and segregate them (and/or the people that use them) 
on special


Tom Harvill
Holland Computing Center
University of Nebraska

More information about the Beowulf mailing list