[Beowulf] Small files
bdobbins at gmail.com
Fri Jun 13 11:23:29 PDT 2014
Just a quick addition to this thread (for now), here's the presentation I
think Killian refers to:
And here's *all* the slides from LUG 2014:
Some of them are incredibly informative, and I have high hopes for some of
the future changes referenced in Andreas Dilger's presentation (
) - specifically, I think the combination of DNE (phase 1 & 2) and small
files on MDTs should help immensely with both IOPS and overall throughput
on those sorts of small-file workloads.
On Fri, Jun 13, 2014 at 2:02 PM, Kilian Cavalotti <
kilian.cavalotti.work at gmail.com> wrote:
> On Fri, Jun 13, 2014 at 7:03 AM, Ellis H. Wilson III <ellis at cse.psu.edu>
> > a) Fix it transparently with automatic policies/FS's in the back-end. (I
> > know of at least one FS that packs small files with metadata
> > on SSDs to expedite small file IOPS, but message me off-list for that as
> > start work for that shop soon and don't want to so blatantly advertise).
> > There are limits to how much these policies/FS's can fix though. Bad I/O
> > will still be Bad I/O after a point.
> Speaking of which, there's this project from Los Alamos, PLFS
> (https://github.com/plfs/plfs-core), that tries to aggregate random
> I/O from applications into sequential operations that better suit the
> underlying parallel filesystem. Never used it myself, but I've seen a
> presentation from Brett Kettering at LUG'14 which outlined nice
> performance improvements for some use cases.
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Beowulf