<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div apple-content-edited="true"><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">On 03 Jul 2014, at 09:48, John Hearns <<a href="mailto:hearnsj@googlemail.com">hearnsj@googlemail.com</a>> wrote:</div></div></div><div><br class="Apple-interchange-newline"><blockquote type="cite">On 03/07/2014, Joseph Landman <<a href="mailto:landman@scalableinformatics.com">landman@scalableinformatics.com</a>> wrote:<br><blockquote type="cite"><br><br><br>Now just get all those writing rfps to stop with the madness. The number<br>and it's pursuit are one of the more significant causes of entropy (as in<br>waste heat generation ) as I have ever seen. A dubious metric at best with<br>extremely limited, if any, correlation to end user realizable performance.<br></blockquote><br><br>Absolutely agree. Application level benchmarks for the codes the<br>system is going to run.<br>And also put that in as a simple acceptance test - does the cluster<br>run code X on N processors in the time T you said it would? Wham bam<br>thankyou.<br><br></blockquote><div>In practice using full application benchmarks may be challenging: When buying </div><div>cutting-edge gear, the vendors usually have rely on extrapolating the performance </div><div>of the application for future hardware. </div><div><br></div><div>The more compilcated the application, the more work it is to extrapolate and </div><div>there is more uncertainity. This in turn will (at least in the current economic climate) </div><div>put most vendors in “risk minimization” mode: They avoid overcommitment and</div><div>have very conservative bids or even fall back to offering “safe" current-generation</div><div>hardware *yawn*. </div><div><br></div><div>Thus, capturing a minimal set of relevant critical performance metrics in synthetic</div><div>benchmarks and simple kernels would be ideal. The downside is that this may not</div><div>always be a simple task either.</div><div><br></div><blockquote type="cite"><br>What are the audiences thoughts on the HPCC benchmarks?<br><br></blockquote><br></div><div>They provide a nice toolkit for the kind of evaluation I mentioned above. For some things</div><div>(like I/O) complementary benchmarks should be used though. </div><div><br></div><div>Having a single number benchmark like HPL is nice for evaluating trends though and</div><div>I’m pretty sure also that the need for having a some sort of top ranking will not </div><div>go away soon. There are a couple of interesting efforts to do this in a more balanced way</div><div>that reflects more the “average" application performance:</div><div style="orphans: auto; widows: auto;"><span style="orphans: 2; text-align: -webkit-auto; widows: 2;">- HPGMG (</span><a href="https://hpgmg.org/" style="orphans: 2; text-align: -webkit-auto; widows: 2;">https://hpgmg.org/</a><span style="orphans: 2; text-align: -webkit-auto; widows: 2;">)</span></div><div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">- HPCG (<a href="https://software.sandia.gov/hpcg/" style="text-align: -webkit-auto;">https://software.sandia.gov/hpcg/</a>) </div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br></div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Here are some interesting new results of how HPCG ranking changes the top system</div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">ranking in HPL: </div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><a href="https://twitter.com/a_z_e_t/status/482062498477383680/photo/1" style="text-align: -webkit-auto;">https://twitter.com/a_z_e_t/status/482062498477383680/photo/1</a></div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br></div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">O-P</div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">-- <br>Olli-Pekka Lehto<br>Development Manager, Computing Platforms </div><div style="orphans: 2; text-align: -webkit-auto; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">CSC - IT Center for Science Ltd.<br>E-Mail: <a href="mailto:olli-pekka.lehto@csc.fi">olli-pekka.lehto@csc.fi</a> // Tel: +358 50 381 8604 // skype: oplehto // twitter: @ople</div></div><div><br></div><br></body></html>