<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix"><br>
On 11/25/2013 07:11 PM, Adam DeConinck wrote:<br>
</div>
<blockquote cite="mid:20131126001130.GB8753@gmail.com" type="cite">
<pre wrap="">-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
</pre>
<blockquote type="cite">
<pre wrap="">4. I went to a BoF on ROI on HPC investment. All the presentations in
the BoF frustrated me. Not because they were poorly done, but because
they tried to measure the value of a cluster by number of papers
published that used that HPC resource. I think that's a crappy, crappy
metric, but haven't been able to come up with a better one myself yet. I
was very vocal with my comments and criticisms of the presentations, so
if any of the presenters are reading this now, I apologize for
hi-jacking your BoF. Getting good ROI on a cluster is close to my heart,
but is also difficult to quantify and measure. I hope I can be part of
the discussion next year.
</pre>
</blockquote>
<pre wrap="">
Do you have any thoughts you can share on what alternative metrics
might look like, even if you can't think of one that's clearly better?</pre>
</blockquote>
<br>
Unfortunately, I have no better suggestions at this time. The best
metric will vary depending on what your priorities and business
goals are. I've had some discussions on different metrics for
different goals, but nothing fruitful/concrete has come out of them.
Too nebulous. <br>
<br>
Many people just want to know if their cluster is being fully
utilized and how full the queue is, under the assumption that if the
existing hardware is at capacity, that's enough justification for
spending more money on more hardware, or proves that the cluster was
needed. However, I think even utilization of an HPC resource can be
misleading, since someone could be doing something stupid in their
codes that might be keeping the processors busy, but isn't really
helping them get a solution any quicker. Also, this doesn't measure
the opportunity cost of researcher swho want to use the resource but
can't because they don't know how, or don't even know it's available
to them. <br>
<br>
Right now, the only answer I can think of regarding system
utilization is "ask your cluster system admin - he sees everything
that goes on, and talks to the users needing support, so he probably
has a good feel for what's going on on your systems." Unfortunately,
that doesn't make for pretty Powerpoint slides. <br>
<br>
<blockquote cite="mid:20131126001130.GB8753@gmail.com" type="cite">
<pre wrap="">
I have no horse in this race as I've been doing industry HPC for the
past few years, but I'm curious what good metrics for ROI on an academic
or lab cluster might be. Total number of papers? Number of
citations after an N-year time window? [shrug]</pre>
</blockquote>
There are two metrics that are often discussed in academia: The
number of papers published based on that research, or the amount of
grant money brought in by researchers using that resource. There is
a paper Amy Apon and other on this topic, titled "High Performance
Computing Instrumentation and Research Productivity in U.S.<br>
Universities":<br>
<br>
<cite class="vurls"><a class="moz-txt-link-freetext" href="http://www.jiti.net/v10/jiti.v10n2.087-098.pd">http://www.jiti.net/v10/jiti.v10n2.087-098.pd</a></cite>f<br>
<h1 style="font-size: 18px;"><br>
</h1>
<blockquote cite="mid:20131126001130.GB8753@gmail.com" type="cite">
<pre wrap="">
ROI measurement can sometimes be difficult even in an industrial or
commercial setting, especially if the HPC resource is used for R&D or
"engineering support" as opposed to something that feeds directly into
the product.</pre>
</blockquote>
<br>
It can be difficult, but I think it's still much easier than in an
academic setting, where many consider the labor of grad students to
be free, and it's difficult to put a price on the value of research
results. <br>
<blockquote cite="mid:20131126001130.GB8753@gmail.com" type="cite">
<pre wrap="">
Cheers,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Darwin)
iQEcBAEBCgAGBQJSk+cxAAoJEGqxms5cZfz0vEcH+gLqL/LmLZgDUcASJGn1U+M0
SbHbwFTbA6DOeMym4gAfvfXrKueabFYA4qHPmAaCoC2XVhCr6vORMLIn27hoA0S7
0dI+hN0H4OUC9u9Xgp9HZtPa2YD1nlYSby6A3D751VDW5m1r4Sn3aafoODyZGkcp
7FyAPhCnK+EgUinICgnfaXVMvCsUJ6cMMi2OicKjLPw85KIkRMfyDbAgh3myE7p2
S6SHKBb3PHzWqIAUZ2iEU+C1XJbwH7bw83ciOJN49T0PPX5+I1Lced76RWcIZC40
YqYdF1kUhBBD7VMonByUBAcCRq7fl9cCLDnEh78z3pZ7+/4dqJcmmMjqyyavrmA=
=xgE9
-----END PGP SIGNATURE-----
_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
<br>
</body>
</html>