[Beowulf] Clearing out scratch space
Ellis H. Wilson III
ellis at ellisv3.com
Tue Jun 12 07:51:54 PDT 2018
On 06/12/2018 10:25 AM, John Hearns via Beowulf wrote:
> > Disclaimer: I work for one such parallel and fast (and often used for
> scratch) company called Panasas.
> Ellis, I know Panasas well of course. You are a great bunch of guys and
> girls, and have pulled my chestnuts from the fire many times (such as
> the plaintive call from the customer - we can't access our data. What
> are all these red lights for - wwe have seen them for weeks. Me - puts
> in call to 800-PANASAS immediately)
> Having been there on my hands and knees installing Panasas systems for
> UK Government defence customers.
> Installed Panasas in Formula 1 where it had a dramatic effect on our
> solver times.
> Also installing the largest Panasas setup in the world at the UK
> Rutherford Appleton Laboratory, which is being used for lcimate resarch
> Been there, seen it, unboxed hundreds of blades.
Very glad you've had such good experiences John. As we like to say
internally, people buy for the filesystem engineering at Panasas, but
stay for the service, and that's still probably giving our architecture
and engineering teams too much credit. Our support team puts the rest
of us to shame.
Returning to the subject at hand, I'm wondering aloud if people prefer
automatic deletion policies for scratch volumes to be incorporated into
the filesystem or not. I ask because while external software is more
flexible in that it should run fine on any POSIX-compliant filesystem,
it has no good way of assuring thousands of deletions per second when a
batch of files age-out aren't interfering with foreground performance.
If instead such aging policies for a scratch volume were more tightly
integrated into the filesystem in question it could balance issuing the
deletes with maintaining a QoS level for foreground performance.
We don't have such a feature, but we have much of the infrastructure to
support a specialized volume type like that without a huge lift. If
people were sufficiently interested in it I could definitely see us
dedicating some engineering resources to it.
Ellis H. Wilson III, Ph.D.
More information about the Beowulf