[Beowulf] Small files
samuel at unimelb.edu.au
Thu Jun 12 23:38:27 PDT 2014
-----BEGIN PGP SIGNED MESSAGE-----
On 12/06/14 05:03, Tom Harvill wrote:
> We've found that a large majority of our files (~40MM of ~50MM)
> are less than 10KB. We believe our filesystem (lustre) is
> bottlenecked with IOPs and locking related to jobs running against
> these files. We have ~700TB usable storage with ~500TB consumed,
> almost all consumption is by a relatively small number of very very
> large files.
Coincidentally I was talking to some people involved in Lustre
yesterday about it and I asked them how well it coped. There was a
bit of shuffling of feet and the answer was along the lines that it
wasn't really their target use case.
Our GPFS filesystem has a lot of fairly small files, but nothing like
the proportion that you are seeing (4/5ths < 10kB).
We have had odd edge cases, like people running OpenFOAM creating a
directory hierarchy of millions of directories, each with 5 files in
them. That was fun...
All the best!
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
-----END PGP SIGNATURE-----
More information about the Beowulf