<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yes.. this is something that has been researched and tested in the laboratory. I don’t know that anyone has actually tried reconfiguring around a damaged piece
of an FPGA, if for no other reason than permanent damage in a reconfigurable FPGA is extremely unusual (and probably hasn’t ever occurred). There are soft upsets in the configuration memory, and the Virtex and Virtex II have a potential failure mode where
an upset in just the wrong place could cause damage (having two logic element outputs fighting each other), but it’s very unlikely.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">There’s a fair amount of test data on radiation behavior (klabs.org or MAPLD are places to look). I’m not sure there’s a failure mechanism (with high enough
probability) that causes a hard failure of just some gates. (These parts are typically latchup-immune, for instance). I suppose some sufficiently high energy particle could damage a few gates permanently. You’d need very high Linear Energy Transfer, though.
There’s a paper by Fuller, et al, out there where they zapped a Virtex with 2068 MeV Au ions, looking to see if latchup could be observed at any LET below 125 MeV-cm^2/mg (this is the upper bound for galactic cosmic rays). No latchup detected. They did
see an increase in current, but it’s because of the configuration upsets causing internal logic contention, and went away when the device was reconfigured. (fluence was 1E7-1E8 ions/cm^2, which is HUGE compared to what you see in real life. There were some
changes in current that stuck around for a few hours, but gradually annealed away)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As far as upsets go, typical predicted upset rates aer on the order of 2 upsets/device day in LEO up to 5.9 upsets/device day in GEO. With flare enhancement,
it’s like 21 upsets/device day for LEO and 81.5 for GEO. (of course, life is better than this.. in most designs, the vast majority of configuration bits are “don’t care”, so you wouldn’t see the upset.. a typical multiplier is 4:1. That is, half an upset/device
day for LEO) (all these are for the XVQR300)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(another source reports a cross section for proton SEU of 5E-13 cm^2/bit.. the device has, say, 6E6 bits, so you can figure out what kind of proton flux you
need to get a given upset rate)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">And, of course, now there’s a rad hardened Virtex 5 available (you too can own one for about $80k/copy).. 1Mrad(Si) total dose, config mem upset rate in GEO
3.8E-10 errors/bit/day. Single Event Functional Interrupt (SEFI) of configuration control logic (this would prevent you from reconfiguring on the fly) in GEO is once every 10,000 years.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So it’s not really clear that you NEED to be able to reconfigure around damage..
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We’ve only been flying Xilinx Virtex parts for long durations since 2005 (Mars Reconnaissance Orbiter) (there might be some other earlier experiments.. CANDOS
used a couple of Virtex II parts on Shuttle was 2 weeks in 2003 and only operated for 10s of hours) We do periodic scrubbing/reloading of the configuration memory, and I’m not sure we even know if there was a transient upset (that is, we don’t read it back,
we just rewrite, blindly). There’s some DoD comm payloads that use Virtex parts, and their mitigation strategy for configuration upsets is to have two devices and ping pong between them.. while chip 1 is being configured, use chip 2, when done, flip, reconfigure
chip 2 and use chip 1.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">When all is said and done, reconfiguring to get around a human coding error is actually much more likely.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Jim Lux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org]
<b>On Behalf Of </b>Nathan Moore<br>
<b>Sent:</b> Wednesday, September 05, 2012 8:24 AM<br>
<b>To:</b> beowulf@beowulf.org<br>
<b>Subject:</b> Re: [Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><br>
> On Tue, 4 Sep 2012, Ellis H. Wilson III wrote:<o:p></o:p></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Which is why I was suggesting that, "Maybe the whole thing is just<o:p></o:p></p>
</div>
<p class="MsoNormal">built, sealed for good, primed with [hydrogen/oil/he/whatever], started,<o:p></o:p></p>
<div>
<p class="MsoNormal">allowed to slowly degrade over time and finally tossed when the still<o:p></o:p></p>
</div>
<p class="MsoNormal">working equipment is sufficiently dated." <o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I remember an "ancient" IBM technical article about the BlueGene, here:
<a href="http://researcher.watson.ibm.com/researcher/files/us-ajayr/SysJ_BlueGene.pdf">
http://researcher.watson.ibm.com/researcher/files/us-ajayr/SysJ_BlueGene.pdf</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In the work (or maybe it was a closely related paper), the authors make the point that as core count increases and feature size decreases, cpu units will have to be fault tolerant, eg if cosmic rays have toasted 10% of your chip's cores,
it should still be able to function. Related, this is one of the great beauties of FPGA's. Jim Lux can probably tell us if this would be real, but it would seem to make sense to program a space probe (ie voyager type) with an FPGA emulated CPU for the sake
of damage survivability. In the worst case that the probe encounters something unpleasant and part of the FPGA is damaged, perhaps the rest of the LUT's in the FPGA could be reprogrammed to produce a less powerful, yet still functional, controller. This
would take the "field programmable" aspect to the device to a new height...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nathan<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>