<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>We also use a "workspace" mechanism to control our scratch
filesystem. It's a home-grown system which works like so:<br>
</p>
<p>0. /scratch filesystem configured with root as its owner and no
other write permission allowed.<br>
</p>
<p>1. User calls mkworkspace which does a setuid, creates the
directory, chowns it to the user, then writes a small record file
about the workspace somewhere outside of it.</p>
<p>2. From cron, rmworkspace runs which reads each record file,
determines which workspaces have expired, then removes the expired
directories and record file.</p>
<p>It works well and we have not had a problem with our scratch
space filling up after 2-3 years of running it. Each compute node
has a local SSD and mkworkspace controls access to that too.
Almost nobody uses this "local scratch" though.</p>
<p>The major downside with this is it that confuses a small portion
of users. They would rather just use mkdir by itself instead of
using our mkworkspace, despite it just being a wrapper for mkdir
with some extra options. We opted not to use prolog/epilog
scripts or other Slurm features to automatically create or remove
workspaces.</p>
<p><br>
</p>
<p>(it should run as a special user instead of root but I would need
to give it CAP_CHOWN to be able to chown and my filesystem doesn't
support Linux capabilities)<br>
</p>
<div class="moz-signature"><tt><br>
Jeff White<br>
HPC Systems Engineer - ITS<br>
Question about or help with Kamiak? Please submit a <a
href="https://hpc.wsu.edu/support/service-requests/">Service
Request</a>.</tt>
</div>
<div class="moz-cite-prefix">On 06/12/2018 02:06 AM, Dmitri Chubarov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANe5L+QBNyfcBipeRUfMok3WXyh8wzOUUu07g+MFywDiXvOf=A@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<div>Hello, John,</div>
<div><br>
</div>
<div>In HLRS they have what they call a Workspace mechanism (<a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__wickie.hlrs.de_platforms_index.php_Workspace-5Fmechanism&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=BuKVxMh1wyVgWXiKoJaXUL7QP0qpxl50rxXlc8JDc84&s=x9rK7qB6v7BhVjwXDATRPqsC57eL1jnCVrBDsSpNtY0&e="
moz-do-not-send="true">https://wickie.hlrs.de/platforms/index.php/Workspace_mechanism</a>)
where each user</div>
<div>creates a scratch directory for their project under
$SCRATCH_ROOT that has end-of-life time encoded in the name
and a symlink to this directory <br>
</div>
<div>in their persistent storage directory tree. A cronjob
enforces the end of life policy.</div>
<div><br>
</div>
<div>One advantage is that it is very easy for the admin to
extend the lifespan when it is absolutely needed. It requires
only renaming one directory to extend, for example, the
lifetime</div>
<div>of millions of files from genomic applications.<br>
</div>
<div> <br>
</div>
<div>Here at Novosibirsk University where users are getting
their resources for free this mechanism has been reimplemented
to ensure that shared storage does not turn into a file
archive. <br>
</div>
<div>The main shared storage is an expensive PanFS system that
is split into two partitions: a larger scratch partitions with
a directory lifetime limit of 90 days and a smaller $HOME
partition.</div>
<div><br>
</div>
<div>Some users in fact are abusing the system by recreating a
new scratch directory every 90 days and copying the data along
effectively creating persistent storage. However most of the
users do evacuate their valuable data on time.<br>
</div>
<div><br>
</div>
<div>Greetings from sunny Siberia,</div>
<div> Dima<br>
</div>
<div>sys and it works by setting draconian limits <br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 12 Jun 2018 at 15:06, John Hearns via
Beowulf <<a href="mailto:beowulf@beowulf.org"
moz-do-not-send="true">beowulf@beowulf.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>In the topic on avoiding fragmentation Chris Samuel
wrote:</div>
<div><br>
</div>
<div>>Our trick in Slurm is to use the slurmdprolog
script to set an XFS project<br>
>quota for that job ID on the per-job directory
(created by a plugin which<br>
>also makes subdirectories there that it maps to /tmp
and /var/tmp for the<br>
>job) on the XFS partition used for local scratch on
the node.<br>
</div>
<div><br>
</div>
<div>I had never thought of that, and it is a very neat
thing to do.</div>
<div>What I would like to discuss is the more general topic
of clearing files from 'fast' storage.</div>
<div>Many sites I have seen have dedicated fast/parallel
storage which is referred to as scratch space.</div>
<div>The intention is to use this scratch space for the
duration of a project, as it is expensive.</div>
<div>However I have often seen that the scratch space i used
as permanent storage, contrary to the intentions of
whoever sized it, paid for it and installed it.<br>
</div>
<div><br>
</div>
<div>I feel that the simplistic 'run a cron job and delete
files older than N days' is outdated.</div>
<div><br>
</div>
<div>My personal take is that heirarchical storage is the
answere, automatically pushing files to slower and cheaper
tiers.</div>
<div><br>
</div>
<div>But the thought struck me - in the Slurm prolog script
create a file called</div>
<div>THESE-FILES-WILL-SELF-DESTRUCT-IN-14-DAYS</div>
<div>Then run a cron job to decrement the figure 14</div>
<div>I guess that doesnt cope with running multiple jobs on
the same data set - but then again running a job marks
that data as 'hot' an dyou reset the timer to 14 days.</div>
<div><br>
</div>
<div>What do most sites do for scratch space?<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit
<a
href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.beowulf.org_mailman_listinfo_beowulf&d=DwMFaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=BuKVxMh1wyVgWXiKoJaXUL7QP0qpxl50rxXlc8JDc84&s=J8PMbp5NubCpV3X9xng3I9DoT0cO1gDRn92UyMrjhZo&e="
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.beowulf.org_mailman_listinfo_beowulf&d=DwIGaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=BuKVxMh1wyVgWXiKoJaXUL7QP0qpxl50rxXlc8JDc84&s=J8PMbp5NubCpV3X9xng3I9DoT0cO1gDRn92UyMrjhZo&e=">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.beowulf.org_mailman_listinfo_beowulf&d=DwIGaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=BuKVxMh1wyVgWXiKoJaXUL7QP0qpxl50rxXlc8JDc84&s=J8PMbp5NubCpV3X9xng3I9DoT0cO1gDRn92UyMrjhZo&e=</a>
</pre>
</blockquote>
<br>
</body>
</html>